title: Tensorflow: completely different results from a same initial guess
body: <p>I am new to Tensorflow. I did a simple multivariate regression with the ordinary GD optimization algorithm. However, I got completely different results when two different variable definitions are applied even with a same initial guess. </p>
  
  <p>What is the difference between the two calculations?  </p>
  
  <p>When I defined variables with:</p>
  
  <pre><code>tau = tf.Variable([0.25, 0.25, 0.25, 0.25], name='parameter', dtype=tf.float64)
  tau = tf.clip_by_value(tau, 0.1, 5.)
  </code></pre>
  
  <p>I got the result below after 10000 epochs.</p>
  
  <p><code>tau= [0.28396885 0.24675105 0.26584612 1.37071573]</code></p>
  
  <p>However, when I defined them as normalized values:</p>
  
  <pre><code>tau_norm = tf.Variable([0.025, 0.025, 0.025, 0.025], name='parameter', dtype=tf.float64)
  tau_norm = tf.clip_by_value(tau_norm, 0.01, 0.5)
  tau_max = 10
  tau = tau_norm*tau_max
  </code></pre>
  
  <p>I got a completely different result after same 10000 epochs:</p>
  
  <p><code>tau= [       nan 0.22451382 2.70862284 1.46199275]</code></p>
  
  <p>I expect these two calculations give a same (or similar enough) result because of the same initial guess. However, that was not what I saw. I wonder what causes this difference. </p>
  
  <p>Here, I use tensorflow-gpu 1.14.0, but GPU is not used for this calculation with:</p>
  
  <pre><code>import os
  os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
  os.environ["CUDA_VISIBLE_DEVICES"]="-1"
  </code></pre>
  
  <hr>
  
  <h3>Updated</h3>
  
  <p>Okay, let me explain with an example with the code adapted from <a href="https://databricks.com/tensorflow/training-and-convergence" rel="nofollow noreferrer">here</a>. 
  I think what I have seen is basically same as below.</p>
  
  <pre><code>import os
  os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
  os.environ["CUDA_VISIBLE_DEVICES"]="-1"
  import tensorflow as tf
  import numpy as np
  
  x = tf.placeholder("float")
  y = tf.placeholder("float")
  w = tf.Variable([1.0, 2.0], name="w")
  
  y_model = tf.multiply(x, w[0]) + w[1]
  error = tf.square(y - y_model)
  train_op = tf.train.GradientDescentOptimizer(0.01).minimize(error)
  
  model = tf.global_variables_initializer()
  
  with tf.Session() as session:
      session.run(model)
      print("Initial guess: ", session.run(w))
      np.random.seed(seed=100)
      for i in range(1000):
          x_value = np.random.rand()
          y_value = x_value * 2 + 6
          session.run(train_op, feed_dict={x: x_value, y: y_value})
  
      w_value = session.run(w)
      print("Predicted model: {a:.3f}x + {b:.3f}".format(a=w_value[0], b=w_value[1]))
  </code></pre>
  
  <p>From the code, I got <code>Predicted model: 2.221x + 5.882</code>. However, when I replaced <code>w</code> with</p>
  
  <pre><code>w_norm = tf.Variable([0.5, 1.0], name = 'w_norm')
  w = w_norm*2.0
  </code></pre>
  
  <p>the result was <code>Predicted model: 2.004x + 5.998</code> even though it has same initial guess (<code>[1. 2.]</code>) and same number of epochs. I do not know what makes this difference. </p>