<p dir="auto"><strong>System information</strong></p>
<p dir="auto">Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Slightly.</p>
<p dir="auto">OS Platform and Distribution : centos6.3</p>
<p dir="auto">TensorFlow installed from: conda</p>
<p dir="auto">TensorFlow version (use command below): v1.8.0(it already has the code TF_CALL_half(REGISTER_BATCH_MATMUL_GPU);)</p>
<p dir="auto">Python version: 3.6</p>
<p dir="auto">CUDA/cuDNN version: 9.2</p>
<p dir="auto">GPU model and memory: V100</p>
<p dir="auto">Exact command to reproduce:</p>
<p dir="auto">with tf.device("/gpu:0"):<br>
a = tf.random_normal(dtype=tf.float16, shape=[5, 2, 3], name='a')<br>
b = tf.random_normal(dtype=tf.float16, shape=[5, 3, 2], name='b')<br>
c = tf.matmul(a, b)<br>
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True, allow_soft_placement=False))<br>
print(sess.run(c).shape)</p>
<p dir="auto">the problem:</p>
<p dir="auto">Source code / logs</p>
<p dir="auto">InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'MatMul_1': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.<br>
Registered kernels:<br>
device='GPU'; T in [DT_DOUBLE]<br>
device='GPU'; T in [DT_FLOAT]<br>
device='GPU'; T in [DT_COMPLEX128]<br>
device='GPU'; T in [DT_COMPLEX64]<br>
device='CPU'; T in [DT_INT32]<br>
device='CPU'; T in [DT_HALF]<br>
device='CPU'; T in [DT_DOUBLE]<br>
device='CPU'; T in [DT_FLOAT]<br>
device='CPU'; T in [DT_COMPLEX128]<br>
device='CPU'; T in [DT_COMPLEX64]</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="     [[Node: MatMul_1 = BatchMatMul[T=DT_HALF, adj_x=false, adj_y=false, _device=&quot;/devic"><pre class="notranslate"><code class="notranslate">     [[Node: MatMul_1 = BatchMatMul[T=DT_HALF, adj_x=false, adj_y=false, _device="/devic
</code></pre></div>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:<br>
MacOS Sierra 12.12.4</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:<br>
pip</li>
<li><strong>TensorFlow version (use command below)</strong>:<br>
1.1.0 (CPU)</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I am trying to implement E2C (available from <a href="https://arxiv.org/pdf/1506.07365.pdf" rel="nofollow">https://arxiv.org/pdf/1506.07365.pdf</a>). Basically it is a neural network that is used for learning a transition model using neural networks. In the training set I have data of the form (X_t, X_t+1) where both X_t and X_t+1 needs to be transformed by an encoding network (e.g. a variational autoencoder). I use the following snippet for creating the encoding network (adapted from <a href="https://github.com/ericjang/e2c">https://github.com/ericjang/e2c</a>):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    def encode(self, x, share=None):
        fc = tf.contrib.layers.fully_connected
        with tf.variable_scope('Encoder', reuse=share):
            l1 = fc(x, 400, weights_initializer=tf.orthogonal_initializer(),
                    activation_fn=tf.nn.relu)
            l2 = fc(l1, 100, weights_initializer=tf.orthogonal_initializer(),
                    activation_fn=tf.nn.relu)
            return l2

    def decode(self, z, share=None):
        fc = tf.contrib.layers.fully_connected
        with tf.variable_scope(&quot;Decoder&quot;, reuse=share):
            l1 = fc(z, 100, weights_initializer=tf.orthogonal_initializer(1.1),
                    activation_fn=tf.nn.relu)
            l2 = fc(l1, 400, weights_initializer=tf.orthogonal_initializer(1.1),
                    activation_fn=tf.nn.relu)

            return fc(l2, self.x_dim,
                      weights_initializer=tf.orthogonal_initializer(1.1),
                      activation_fn=tf.nn.sigmoid)"><pre class="notranslate">    <span class="pl-k">def</span> <span class="pl-en">encode</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>, <span class="pl-s1">share</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-s1">fc</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">layers</span>.<span class="pl-s1">fully_connected</span>
        <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">variable_scope</span>(<span class="pl-s">'Encoder'</span>, <span class="pl-s1">reuse</span><span class="pl-c1">=</span><span class="pl-s1">share</span>):
            <span class="pl-s1">l1</span> <span class="pl-c1">=</span> <span class="pl-en">fc</span>(<span class="pl-s1">x</span>, <span class="pl-c1">400</span>, <span class="pl-s1">weights_initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">orthogonal_initializer</span>(),
                    <span class="pl-s1">activation_fn</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">relu</span>)
            <span class="pl-s1">l2</span> <span class="pl-c1">=</span> <span class="pl-en">fc</span>(<span class="pl-s1">l1</span>, <span class="pl-c1">100</span>, <span class="pl-s1">weights_initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">orthogonal_initializer</span>(),
                    <span class="pl-s1">activation_fn</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">relu</span>)
            <span class="pl-k">return</span> <span class="pl-s1">l2</span>

    <span class="pl-k">def</span> <span class="pl-en">decode</span>(<span class="pl-s1">self</span>, <span class="pl-s1">z</span>, <span class="pl-s1">share</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-s1">fc</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">layers</span>.<span class="pl-s1">fully_connected</span>
        <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">variable_scope</span>(<span class="pl-s">"Decoder"</span>, <span class="pl-s1">reuse</span><span class="pl-c1">=</span><span class="pl-s1">share</span>):
            <span class="pl-s1">l1</span> <span class="pl-c1">=</span> <span class="pl-en">fc</span>(<span class="pl-s1">z</span>, <span class="pl-c1">100</span>, <span class="pl-s1">weights_initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">orthogonal_initializer</span>(<span class="pl-c1">1.1</span>),
                    <span class="pl-s1">activation_fn</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">relu</span>)
            <span class="pl-s1">l2</span> <span class="pl-c1">=</span> <span class="pl-en">fc</span>(<span class="pl-s1">l1</span>, <span class="pl-c1">400</span>, <span class="pl-s1">weights_initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">orthogonal_initializer</span>(<span class="pl-c1">1.1</span>),
                    <span class="pl-s1">activation_fn</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">relu</span>)

            <span class="pl-k">return</span> <span class="pl-en">fc</span>(<span class="pl-s1">l2</span>, <span class="pl-s1">self</span>.<span class="pl-s1">x_dim</span>,
                      <span class="pl-s1">weights_initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">orthogonal_initializer</span>(<span class="pl-c1">1.1</span>),
                      <span class="pl-s1">activation_fn</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">sigmoid</span>)</pre></div>
<p dir="auto">Then I would use something like</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="h_enc_t = encoder(X_t)
h_enc_t_next = encoder(X_{t+1}, share=True)"><pre class="notranslate"><span class="pl-s1">h_enc_t</span> <span class="pl-c1">=</span> <span class="pl-en">encoder</span>(<span class="pl-v">X_t</span>)
<span class="pl-s1">h_enc_t_next</span> <span class="pl-c1">=</span> <span class="pl-en">encoder</span>(<span class="pl-v">X_</span>{<span class="pl-s1">t</span><span class="pl-c1">+</span><span class="pl-c1">1</span>}, <span class="pl-s1">share</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</pre></div>
<p dir="auto">to create the encoded output for the model.</p>
<p dir="auto">The problem is that when visualizing this on Tensorboard, while it is sharing the variables by setting <code class="notranslate">share=True</code> for the variable scope, on the graph visulisation you will have <code class="notranslate">Encoder</code> and <code class="notranslate">Encoder_1</code> instead of just a <code class="notranslate">Decoder</code> scope. Of course they took different input since we need to transform X_t and X_t+1, but shouldn't the network be wrapped in the same scope since underneath we are reusing the same weights? I wonder if it is a feature to have <code class="notranslate">Encoder_1</code> and <code class="notranslate">Encoder</code> separately or it is a limitation of the variable scoping. The problem is illustrated in the screenshot below, you will see duplicates for 'Encoder' 'SampleQPhi" etc:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6040760/25559064/98fecc5a-2d2b-11e7-8669-00b1227abf17.png"><img src="https://cloud.githubusercontent.com/assets/6040760/25559064/98fecc5a-2d2b-11e7-8669-00b1227abf17.png" alt="graph-run" style="max-width: 100%;"></a></p>
<p dir="auto">However, I would expect something like this (as appeared in the paper) to be a more reasonable visualization (h_enc) with input x_t and x_t+1 are the same network:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6040760/25565322/fc04ab8a-2dbc-11e7-9876-5e823a3bf47b.png"><img width="546" alt="screenshot 2017-04-30 15 51 01" src="https://cloud.githubusercontent.com/assets/6040760/25565322/fc04ab8a-2dbc-11e7-9876-5e823a3bf47b.png" style="max-width: 100%;"></a></p>
<p dir="auto">Many thanks in advance!</p>