<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04</li>
<li>TensorFlow installed from (source or binary): source</li>
<li>TensorFlow version: 1.13.1</li>
<li>Python version: 2.7</li>
<li>Bazel version (if compiling from source): 0.21.0</li>
<li>GCC/Compiler version (if compiling from source): 7.3.0</li>
<li>CUDA/cuDNN version: 10.0</li>
<li>GPU model and memory: GeForce GTX 970</li>
</ul>
<p dir="auto"><strong>Describe the problem</strong></p>
<p dir="auto">The set of CUDA compute capabilities (CCC from now on) is set as {3.0, 5.2} during configuration. File <code class="notranslate">.tf_configure.bazelrc</code> contains <code class="notranslate">build --action_env TF_CUDA_COMPUTE_CAPABILITIES="3.0,5.2"</code> (full file: <a href="https://github.com/tensorflow/tensorflow/files/3043058/tf_configure.bazelrc.txt">tf_configure.bazelrc.txt</a>). After transfering the resulting pip package to a PC with CCC 3.0, tensorflow will reject a GPU with CCC lower than 3.5.</p>
<p dir="auto"><strong>Provide the exact sequence of commands / steps that you executed before running into the problem</strong></p>
<ol dir="auto">
<li><code class="notranslate">./configure</code> (resulting in <a href="https://github.com/tensorflow/tensorflow/files/3043058/tf_configure.bazelrc.txt">tf_configure.bazelrc.txt</a>)</li>
<li><code class="notranslate">bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package</code></li>
<li><code class="notranslate">./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg</code></li>
<li>move <code class="notranslate">/tmp/tensorflow_pkg/tensorflow-1.13.1-cp27-cp27mu-linux_x86_64.whl</code> to other PC with CCC 3.0</li>
</ol>
<p dir="auto">Error message occurs when applying L2 loss weight decay with 1080ti gpu activated<br>
No Error message occurs when applying weight decay with only cpu running<br>
The error message always come out if training with all the Optimizer in anaconda, win10, gpu 1080ti, tensorflow 1.6.0. CUDA 9 CUDNN 7.0</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;train_ADAM_add1CNN.py&quot;, line 89, in &lt;module&gt;
    _ = sess.run([train_step_adam],feed_dict={img_in:X_mb, label_gt:y_gt})
  File &quot;D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py&quot;, line 905, in run
    run_metadata_ptr)
  File &quot;D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py&quot;, line 1137, in _run
    feed_dict_tensor, options, run_metadata)
  File &quot;D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py&quot;, line 1355, in _do_run
    options, run_metadata)
  File &quot;D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py&quot;, line 1374, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: AttrValue must not have reference type value of float_ref
         for attr 'tensor_type'"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "train_ADAM_add1CNN.py", line 89, in &lt;module&gt;
    _ = sess.run([train_step_adam],feed_dict={img_in:X_mb, label_gt:y_gt})
  File "D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py", line 905, in run
    run_metadata_ptr)
  File "D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py", line 1137, in _run
    feed_dict_tensor, options, run_metadata)
  File "D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py", line 1355, in _do_run
    options, run_metadata)
  File "D:\ProgramData\Anaconda4\envs\tff\lib\site-packages\tensorflow\python\client\session.py", line 1374, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: AttrValue must not have reference type value of float_ref
         for attr 'tensor_type'
</code></pre></div>
<p dir="auto">Here is the example code that can duplicate the issue:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="...
def optimize_adam(loss, learning_rate, LRdecaysteps, LRdecayrate):
    decay_learning_rate = tf.train.exponential_decay(learning_rate, global_step, LRdecaysteps, LRdecayrate, staircase=True)
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        return tf.train.AdamOptimizer(decay_learning_rate).minimize(loss, global_step=global_step), decay_learning_rate

regularization_loss = tf.add_n(tf.losses.get_regularization_losses())
total_loss = total_loss + regularization_loss
train_step_adam,decay_learning_rate = optimize_adam(total_loss,learning_rate,LRdecaysteps,LRdecayrate)
..."><pre class="notranslate"><code class="notranslate">...
def optimize_adam(loss, learning_rate, LRdecaysteps, LRdecayrate):
    decay_learning_rate = tf.train.exponential_decay(learning_rate, global_step, LRdecaysteps, LRdecayrate, staircase=True)
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        return tf.train.AdamOptimizer(decay_learning_rate).minimize(loss, global_step=global_step), decay_learning_rate

regularization_loss = tf.add_n(tf.losses.get_regularization_losses())
total_loss = total_loss + regularization_loss
train_step_adam,decay_learning_rate = optimize_adam(total_loss,learning_rate,LRdecaysteps,LRdecayrate)
...
</code></pre></div>
<p dir="auto">it can be run with :<br>
session_conf = tf.ConfigProto(<br>
device_count={'CPU' : 1, 'GPU' : 0},<br>
allow_soft_placement=True,<br>
log_device_placement=False<br>
)</p>
<p dir="auto">But it will come out this error message with:<br>
session_conf = tf.ConfigProto(<br>
device_count={'CPU' : 1, 'GPU' : 1},<br>
allow_soft_placement=True,<br>
log_device_placement=False<br>
)</p>