<p dir="auto">I am glad to see the newly added einsum function. The documentation claims that its usage is the same as numpy. However, it can do almost nothing as compared to numpy. For example, it only supports subscripts in the form of '<em>-&gt;</em>'. Unfortunately, even matrix transpose does not work, i.e., 'ij-&gt;ji'.<br>
Numpy works:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; A
array([[ 0.3828997 , -0.39114848, -0.09727838, -0.20430113],
       [ 0.48020577, -0.47122706,  0.42830791,  0.25665744],
       [-0.30885863,  0.21669025,  0.31648793,  0.22417514],
       [ 0.32505724,  0.30478035,  0.48655034,  0.20040547]])
&gt;&gt;&gt; einsum('ij-&gt;ji',A)
array([[ 0.3828997 ,  0.48020577, -0.30885863,  0.32505724],
       [-0.39114848, -0.47122706,  0.21669025,  0.30478035],
       [-0.09727838,  0.42830791,  0.31648793,  0.48655034],
       [-0.20430113,  0.25665744,  0.22417514,  0.20040547]])"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; A
array([[ 0.3828997 , -0.39114848, -0.09727838, -0.20430113],
       [ 0.48020577, -0.47122706,  0.42830791,  0.25665744],
       [-0.30885863,  0.21669025,  0.31648793,  0.22417514],
       [ 0.32505724,  0.30478035,  0.48655034,  0.20040547]])
&gt;&gt;&gt; einsum('ij-&gt;ji',A)
array([[ 0.3828997 ,  0.48020577, -0.30885863,  0.32505724],
       [-0.39114848, -0.47122706,  0.21669025,  0.30478035],
       [-0.09727838,  0.42830791,  0.31648793,  0.48655034],
       [-0.20430113,  0.25665744,  0.22417514,  0.20040547]])
</code></pre></div>
<p dir="auto">Tensorflow does not work:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pseudo-code:
M_ = tf.Variable(tf.random_normal([4,4]))
N_ = tf.einsum('ij-&gt;ji',M_)              
print [M_, N_]

output:
[array([[ 0.80474716, -1.38590837, -0.3379252 , -1.24965811],
       [ 2.57852983,  0.05492432,  0.23039417, -0.74263287],
       [-2.42627382,  1.70774114,  1.19503212,  0.43006262],
       [-1.04652011, -0.32753903, -1.26430523,  0.8810069 ]], dtype=float32), 
array([[ 0.80474716, -1.38590837, -0.3379252 , -1.24965811],
       [ 2.57852983,  0.05492432,  0.23039417, -0.74263287],
       [-2.42627382,  1.70774114,  1.19503212,  0.43006262],
       [-1.04652011, -0.32753903, -1.26430523,  0.8810069 ]], dtype=float32)]"><pre class="notranslate"><code class="notranslate">pseudo-code:
M_ = tf.Variable(tf.random_normal([4,4]))
N_ = tf.einsum('ij-&gt;ji',M_)              
print [M_, N_]

output:
[array([[ 0.80474716, -1.38590837, -0.3379252 , -1.24965811],
       [ 2.57852983,  0.05492432,  0.23039417, -0.74263287],
       [-2.42627382,  1.70774114,  1.19503212,  0.43006262],
       [-1.04652011, -0.32753903, -1.26430523,  0.8810069 ]], dtype=float32), 
array([[ 0.80474716, -1.38590837, -0.3379252 , -1.24965811],
       [ 2.57852983,  0.05492432,  0.23039417, -0.74263287],
       [-2.42627382,  1.70774114,  1.19503212,  0.43006262],
       [-1.04652011, -0.32753903, -1.26430523,  0.8810069 ]], dtype=float32)]
</code></pre></div>
<p dir="auto">I want to multiply a matrix with every frame vector in every batch. Or similar operations which can be done by a simple tensor product. It seems that I still have to duplicate the matrix so many times and perform a batch_matmul, which is very inconvenient and slow and memory consuming.</p>
<p dir="auto">I suggest tensorflow to implement either the tensordot or einsum function which can perform tensor product.</p>
<p dir="auto">It is quite a shame that tensorflow cannot even perform basic tensor product so far :(</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: binary</li>
<li><strong>TensorFlow version (use command below)</strong>: 0.12.1</li>
<li><strong>Python version</strong>: 2.7.10</li>
<li><strong>Bazel version (if compiling from source)</strong>: None</li>
<li><strong>CUDA/cuDNN version</strong>: 5.1</li>
<li><strong>GPU model and memory</strong>: 1080-Ti</li>
<li><strong>Exact command to reproduce</strong>: None</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I'm trying to use Tensorflow in conjunction with PyTorch (I built the model in Tensorflow to generate vector representations and PyTorch trains on top of those). However, the problem is that PyTorch runs out of memory because TF will replicate the model in ALL available CUDA devices. In this case CUDA_VISIBLE_DEVICES is not helpful, and I tried GPU device tf.device("/gpu:0") but Tensorflow still fills up all GPUs' memory.</p>
<p dir="auto">Is there some way to actually limit Tensorflow's GPU usage to one and free up the other for other DL libraries like PyTorch or Theano?</p>