Answers.Body: <p>After 100 steps learning rate will switch from 1e-2 to 1e-24.</p>
  <p>What exactly is the problem here? Are you confusing steps with epochs?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-16T09:57:12.997Z
Answers.Text: Are you sure? I've tested this. With 1e-24 learning is completly stalled. However, after 100 steps learning just continues, implying that we are still learning with 1e-2. I'm glad you think it's correct but It does not seem to work
Answers.UserDisplayName: nammerkage
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-16T09:50:16.483Z
Answers.LastActivityDate: 2021-05-16T09:50:16.483Z
Answers.UserDisplayName: spb
Body: <p>I'm trying to manipulate the learning rate with tf PiecewiseConstantDecay.
  I can easily check if the algorithm switches learning rate values, because one rate is extremely low 1e-20 !!
  However, NO setting of the &quot;boundaries&quot; causes the algorithm to switch learning rate... What am I doing wrong?</p>
  <pre><code>step = tf.Variable(0, trainable=False)
  boundaries = [100]
  values = [1e2, 1e-20]
  
  schedule = tf.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)
  lr = 1e-4 * schedule(step)
  
  optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
  model.compile(optimizer=optimizer, loss='mean_squared_error')
  
  history = model.fit(x = x_train, y = y_train, validation_data=(x_val, y_val), epochs=100,batch_size=32)
  </code></pre>
  <p>Would love your input.</p>
  
Tags: <neural-networks><machine-learning>
Title: Tensforflow schedule - does not change boundaries
UserDisplayName: nammerkage
ThreadText: # Tensforflow schedule - does not change boundaries
  
  (Asked by: username_0 on 2021-05-16)
  
  I'm trying to manipulate the learning rate with tf PiecewiseConstantDecay.
  I can easily check if the algorithm switches learning rate values, because one rate is extremely low 1e-20 !!
  However, NO setting of the "boundaries" causes the algorithm to switch learning rate... What am I doing wrong?
  step = tf.Variable(0, trainable=False)
  boundaries = [100]
  values = [1e2, 1e-20]
  
  schedule = tf.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)
  lr = 1e-4 * schedule(step)
  
  optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
  model.compile(optimizer=optimizer, loss='mean_squared_error')
  
  history = model.fit(x = x_train, y = y_train, validation_data=(x_val, y_val), epochs=100,batch_size=32)
  
  Would love your input.
  
  **Tags:** neural-networks, machine-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-16. Score: 1
  
  After 100 steps learning rate will switch from 1e-2 to 1e-24.
  
  What exactly is the problem here? Are you confusing steps with epochs?
  
  > **Comments:**
  > **username_0**: Are you sure? I've tested this. With 1e-24 learning is completly stalled. However, after 100 steps learning just continues, implying that we are still learning with 1e-2. I'm glad you think it's correct but It does not seem to work  
  
  ---
  
  