<p dir="auto">The current Java API's <code class="notranslate">Tensor.create(Object)</code> is really slow - for a batch of 128 images of size 224x224x3 it's taking around 1.5seconds. To put this into perspective <code class="notranslate">runner.run()</code> with that data and an InceptionV3 graph took below 1second so data prep is x1.5 of the runtime here (for a batch of 32 images it's around 0.35-0.45sec).</p>
<p dir="auto">Is this working as intended? When running the Python code (using simple <code class="notranslate">sess.run(fetches, feed_dict=feed_dict)</code>) with which the graph meta file was generated (TF 1.0.1) and feeding a Python array I don't see such hiccups, the speed is the same as the Java <code class="notranslate">runner.run()</code>.</p>
<p dir="auto">Might it be because of build flags used, maybe I'm missing some optimizations?</p>
<p dir="auto">For now this small part is killing the whole performance, bringing it down from 130obs/sec (<code class="notranslate">runner.run()</code> time) to about ~45obs/sec (Tensor.create+run()).</p>
<p dir="auto">A bit of a sidenote, the performance page states:</p>
<blockquote>
<p dir="auto">This will result in poor performance.<br>
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})</p>
</blockquote>
<p dir="auto">But currently there's no other way to feed data from the Java API, right? A queue (able to read from a file and from memory, i.e. from a Java structure) would be amazing.</p>
<h3 dir="auto">Jar build command</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="export CC=&quot;/usr/bin/gcc&quot;
export CXX=&quot;/usr/bin/g++&quot;
export TF_NEED_CUDA=1
export GCC_HOST_COMPILER_PATH=$CC
export BUILDFLAGS=&quot;--config=cuda --copt=-m64 --linkopt=-m64 --copt=-march=native&quot;

bazel build -c opt \
  //tensorflow/java:tensorflow \
  //tensorflow/java:libtensorflow_jni \
  $BUILDFLAGS --spawn_strategy=standalone --genrule_strategy=standalone"><pre class="notranslate"><code class="notranslate">export CC="/usr/bin/gcc"
export CXX="/usr/bin/g++"
export TF_NEED_CUDA=1
export GCC_HOST_COMPILER_PATH=$CC
export BUILDFLAGS="--config=cuda --copt=-m64 --linkopt=-m64 --copt=-march=native"

bazel build -c opt \
  //tensorflow/java:tensorflow \
  //tensorflow/java:libtensorflow_jni \
  $BUILDFLAGS --spawn_strategy=standalone --genrule_strategy=standalone
</code></pre></div>
<h3 dir="auto">Environment info</h3>
<p dir="auto"><strong>OS:</strong> Ubuntu 16.04<br>
<strong>GPU:</strong> GPU TITAN X (Pascal) 12GB<br>
<strong>CPU:</strong> Intel® Xeon® Processor E5-2630 v4 10core<br>
<strong>GPU Drivers:</strong><br>
NVidia CUDA Driver Version: 375.39<br>
CUDNN 5.1.5<br>
CUDA 8<br>
<strong>Tensorflow version:</strong> JAR file built from current master (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/c25ecb53ae0a8b3ebe4b30238433a7588dbe8537/hovercard" href="https://github.com/tensorflow/tensorflow/commit/c25ecb53ae0a8b3ebe4b30238433a7588dbe8537"><tt>c25ecb5</tt></a>)</p>
<h3 dir="auto">Example</h3>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="public void test() {
  Random r = new Random();
  int imageSize = 224 * 224 * 3;
  int batch = 128;
  float[][] input = new float[batch][imageSize];
  for(int i = 0; i &lt; batch; i++) {
    for(int j = 0; j &lt; imageSize; j++) {
      input[i][j] = r.nextFloat();
    }
  }

  long start = System.nanoTime();
  Tensor.create(input);
  long end = System.nanoTime();
  // Around 1.5sec
  System.out.println(&quot;Took: &quot; + (end - start));
}"><pre class="notranslate"><span class="pl-k">public</span> <span class="pl-smi">void</span> <span class="pl-s1">test</span>() {
  <span class="pl-smi">Random</span> <span class="pl-s1">r</span> = <span class="pl-k">new</span> <span class="pl-smi">Random</span>();
  <span class="pl-smi">int</span> <span class="pl-s1">imageSize</span> = <span class="pl-c1">224</span> * <span class="pl-c1">224</span> * <span class="pl-c1">3</span>;
  <span class="pl-smi">int</span> <span class="pl-s1">batch</span> = <span class="pl-c1">128</span>;
  <span class="pl-smi">float</span>[][] <span class="pl-s1">input</span> = <span class="pl-k">new</span> <span class="pl-smi">float</span>[<span class="pl-s1">batch</span>][<span class="pl-s1">imageSize</span>];
  <span class="pl-k">for</span>(<span class="pl-smi">int</span> <span class="pl-s1">i</span> = <span class="pl-c1">0</span>; <span class="pl-s1">i</span> &lt; <span class="pl-s1">batch</span>; <span class="pl-s1">i</span>++) {
    <span class="pl-k">for</span>(<span class="pl-smi">int</span> <span class="pl-s1">j</span> = <span class="pl-c1">0</span>; <span class="pl-s1">j</span> &lt; <span class="pl-s1">imageSize</span>; <span class="pl-s1">j</span>++) {
      <span class="pl-s1">input</span>[<span class="pl-s1">i</span>][<span class="pl-s1">j</span>] = <span class="pl-s1">r</span>.<span class="pl-en">nextFloat</span>();
    }
  }

  <span class="pl-smi">long</span> <span class="pl-s1">start</span> = <span class="pl-smi">System</span>.<span class="pl-en">nanoTime</span>();
  <span class="pl-smi">Tensor</span>.<span class="pl-en">create</span>(<span class="pl-s1">input</span>);
  <span class="pl-smi">long</span> <span class="pl-s1">end</span> = <span class="pl-smi">System</span>.<span class="pl-en">nanoTime</span>();
  <span class="pl-c">// Around 1.5sec</span>
  <span class="pl-smi">System</span>.<span class="pl-s1">out</span>.<span class="pl-en">println</span>(<span class="pl-s">"Took: "</span> + (<span class="pl-s1">end</span> - <span class="pl-s1">start</span>));
}</pre></div>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): <strong>yes</strong></li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): <strong>OS X El Capitan 10.11.6 (15G22010)</strong></li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: <strong>N/A</strong></li>
<li>TensorFlow installed from (source or binary): <strong>binary (anaconda)</strong><br>
output from <code class="notranslate">conda list tensorflow</code>:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tensorflow                1.12.0          mkl_py36h2b2bbaf_0  
tensorflow-base           1.12.0          mkl_py36h70e0e9a_0 "><pre class="notranslate"><code class="notranslate">tensorflow                1.12.0          mkl_py36h2b2bbaf_0  
tensorflow-base           1.12.0          mkl_py36h70e0e9a_0 
</code></pre></div>
<ul dir="auto">
<li>TensorFlow version (use command below): <strong>b'unknown' 1.12.0</strong></li>
<li>Python version:<br>
<strong>Python 3.6.6 |Anaconda custom (64-bit)| (default, Jun 28 2018, 11:07:29) **<br>
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin</strong></li>
<li>Bazel version (if compiling from source): <strong>N/A</strong></li>
<li>GCC/Compiler version (if compiling from source): <strong>N/A</strong></li>
<li>CUDA/cuDNN version: <strong>N/A</strong></li>
<li>GPU model and memory: <strong>N/A</strong></li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I get the following error message only under <strong>specific circumstances</strong> (see code below):</p>
<blockquote>
<p dir="auto">OMP: Error <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="115922135" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/15" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/15/hovercard" href="https://github.com/tensorflow/tensorflow/issues/15">#15</a>: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.<br>
OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it candegrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see <a href="http://www.intel.com/software/products/support/" rel="nofollow">http://www.intel.com/software/products/support/</a>.<br>
Abort trap: 6</p>
</blockquote>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
No error message</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np

def get_dense(input, widths, activations):
    assert len(widths) == len(activations)
    output = input
    for i, (w, a) in enumerate(zip(widths, activations)):
        output = tf.layers.dense(output, units=w, activation=a)
    return output

nn = lambda input, depth, width: get_dense(
        input,
        [width      for _ in range(depth - 1)] + [3   ],
        [tf.nn.relu for _ in range(depth - 1)] + [None]
    )

tf.reset_default_graph()

# Works fine when N = 8192,
# but breaks with N = 8193:
N = 8193
n = 100

X = np.random.normal(loc=0.0, scale=1., size=(N, 1)).astype(np.float32)
nx = X.shape[1]
Y = X**2

XY = np.concatenate([X, Y], axis=1)

ds = tf.data.Dataset.from_tensor_slices(XY).batch(n).make_one_shot_iterator().get_next()

pred = nn(ds[:,:nx], 2, 10)
loss = tf.reduce_mean((pred - ds[:,nx:])**2)
op = tf.train.RMSPropOptimizer(0.0005).minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    try:
        for i in range(20):
            print(i)
            _, l1 = sess.run([op, loss])
            print(l1)
    except tf.errors.OutOfRangeError:
        pass

print(&quot;Done&quot;)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-k">def</span> <span class="pl-en">get_dense</span>(<span class="pl-s1">input</span>, <span class="pl-s1">widths</span>, <span class="pl-s1">activations</span>):
    <span class="pl-k">assert</span> <span class="pl-en">len</span>(<span class="pl-s1">widths</span>) <span class="pl-c1">==</span> <span class="pl-en">len</span>(<span class="pl-s1">activations</span>)
    <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">input</span>
    <span class="pl-k">for</span> <span class="pl-s1">i</span>, (<span class="pl-s1">w</span>, <span class="pl-s1">a</span>) <span class="pl-c1">in</span> <span class="pl-en">enumerate</span>(<span class="pl-en">zip</span>(<span class="pl-s1">widths</span>, <span class="pl-s1">activations</span>)):
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">layers</span>.<span class="pl-en">dense</span>(<span class="pl-s1">output</span>, <span class="pl-s1">units</span><span class="pl-c1">=</span><span class="pl-s1">w</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s1">a</span>)
    <span class="pl-k">return</span> <span class="pl-s1">output</span>

<span class="pl-s1">nn</span> <span class="pl-c1">=</span> <span class="pl-k">lambda</span> <span class="pl-s1">input</span>, <span class="pl-s1">depth</span>, <span class="pl-s1">width</span>: <span class="pl-en">get_dense</span>(
        <span class="pl-s1">input</span>,
        [<span class="pl-s1">width</span>      <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">depth</span> <span class="pl-c1">-</span> <span class="pl-c1">1</span>)] <span class="pl-c1">+</span> [<span class="pl-c1">3</span>   ],
        [<span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">relu</span> <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">depth</span> <span class="pl-c1">-</span> <span class="pl-c1">1</span>)] <span class="pl-c1">+</span> [<span class="pl-c1">None</span>]
    )

<span class="pl-s1">tf</span>.<span class="pl-en">reset_default_graph</span>()

<span class="pl-c"># Works fine when N = 8192,</span>
<span class="pl-c"># but breaks with N = 8193:</span>
<span class="pl-v">N</span> <span class="pl-c1">=</span> <span class="pl-c1">8193</span>
<span class="pl-s1">n</span> <span class="pl-c1">=</span> <span class="pl-c1">100</span>

<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">normal</span>(<span class="pl-s1">loc</span><span class="pl-c1">=</span><span class="pl-c1">0.0</span>, <span class="pl-s1">scale</span><span class="pl-c1">=</span><span class="pl-c1">1.</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-v">N</span>, <span class="pl-c1">1</span>)).<span class="pl-en">astype</span>(<span class="pl-s1">np</span>.<span class="pl-s1">float32</span>)
<span class="pl-s1">nx</span> <span class="pl-c1">=</span> <span class="pl-v">X</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">1</span>]
<span class="pl-v">Y</span> <span class="pl-c1">=</span> <span class="pl-v">X</span><span class="pl-c1">**</span><span class="pl-c1">2</span>

<span class="pl-v">XY</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">concatenate</span>([<span class="pl-v">X</span>, <span class="pl-v">Y</span>], <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)

<span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">data</span>.<span class="pl-v">Dataset</span>.<span class="pl-en">from_tensor_slices</span>(<span class="pl-v">XY</span>).<span class="pl-en">batch</span>(<span class="pl-s1">n</span>).<span class="pl-en">make_one_shot_iterator</span>().<span class="pl-en">get_next</span>()

<span class="pl-s1">pred</span> <span class="pl-c1">=</span> <span class="pl-en">nn</span>(<span class="pl-s1">ds</span>[:,:<span class="pl-s1">nx</span>], <span class="pl-c1">2</span>, <span class="pl-c1">10</span>)
<span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>((<span class="pl-s1">pred</span> <span class="pl-c1">-</span> <span class="pl-s1">ds</span>[:,<span class="pl-s1">nx</span>:])<span class="pl-c1">**</span><span class="pl-c1">2</span>)
<span class="pl-s1">op</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">RMSPropOptimizer</span>(<span class="pl-c1">0.0005</span>).<span class="pl-en">minimize</span>(<span class="pl-s1">loss</span>)

<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
    <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">tf</span>.<span class="pl-en">global_variables_initializer</span>())
    <span class="pl-k">try</span>:
        <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">20</span>):
            <span class="pl-en">print</span>(<span class="pl-s1">i</span>)
            <span class="pl-s1">_</span>, <span class="pl-s1">l1</span> <span class="pl-c1">=</span> <span class="pl-s1">sess</span>.<span class="pl-en">run</span>([<span class="pl-s1">op</span>, <span class="pl-s1">loss</span>])
            <span class="pl-en">print</span>(<span class="pl-s1">l1</span>)
    <span class="pl-k">except</span> <span class="pl-s1">tf</span>.<span class="pl-s1">errors</span>.<span class="pl-v">OutOfRangeError</span>:
        <span class="pl-k">pass</span>

<span class="pl-en">print</span>(<span class="pl-s">"Done"</span>)</pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
I would consider this an anaconda problem if only it happened at all times. However, what I find weird, and why I suspect it might be a tensorflow issue, is that it only occurs when input dataset size is large enough. In this example the error pops up when N is 8193 or higher, no error when N is 8192 or lower. This "threshold" value is different in my original project where I first faced the problem - there it happened when dataset array length was 100001 (100k+1) or higher, while working fine with 100000 (100k) long dataset. Apologies if this is irrelevant to tensorflow and indeed an anaconda issue.</p>
<p dir="auto">P.S.<br>
Adding KMP_DUPLICATE_LIB_OK=TRUE does silence the error.</p>