<p>There are a few things you can play with:</p>

<ul>
<li><p>Try reducing the learning rate, or increasing decay.</p></li>
<li><p>Try using regularization(L1/L2 or dropout)</p></li>
<li><p>Try using momentum(your model may be stuck in a local minima)</p></li>
<li><p>Adjust other hyperparams(nodes, layers, batch size, etc.)</p></li>
</ul>

<p>Unless you have some knowledge about the specific cause of high loss variance, the above steps in some amount should get you where you need to go. </p>

CC BY-SA 4.0
2019-12-02T18:37:11.787Z
How to reduce variance with fixed hyperparameters? That was the question.
dereks
CC BY-SA 4.0
2019-12-01T21:59:00.560Z
2019-12-01T21:59:00.560Z
hisairnessag3
<p>I know that stochastic gradient descent always gives different results. What are the best practices to reduce this variance today?
I tried to predict simple function with two different approaches and every time I train them I see very different results.</p>

<p>Input data:</p>

<pre><code>def plot(model_out):
  fig, ax = plt.subplots()
  ax.grid(True, which='both')
  ax.axhline(y=0, color='k', linewidth=1)
  ax.axvline(x=0, color='k', linewidth=1)

  ax.plot(x_line, y_line, c='g', linewidth=1)
  ax.scatter(inputs, targets, c='b', s=8)
  ax.scatter(inputs, model_out, c='r', s=8)

a = 5.0; b = 3.0; x_left, x_right = -16., 16.
NUM_EXAMPLES = 200
noise   = tf.random.normal((NUM_EXAMPLES,1))

inputs  = tf.random.uniform((NUM_EXAMPLES,1), x_left, x_right)
targets = a * tf.sin(inputs) + b + noise
x_line  = tf.linspace(x_left, x_right, 500)
y_line  = a * tf.sin(x_line) + b
</code></pre>

<p>Keras training:</p>

<pre><code>model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(50, activation='relu', input_shape=(1,)))
model.add(tf.keras.layers.Dense(50, activation='relu'))
model.add(tf.keras.layers.Dense(1))

model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam(0.01))
model.fit(inputs, targets, batch_size=200, epochs=2000, verbose=0)

print(model.evaluate(inputs, targets, verbose=0))
plot(model.predict(inputs))
</code></pre>

<p><a href="https://i.sstatic.net/Apel6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Apel6.png" alt="enter image description here"></a></p>

<p>Manual training:</p>

<pre><code>model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(50, activation='relu', input_shape=(1,)))
model.add(tf.keras.layers.Dense(50, activation='relu'))
model.add(tf.keras.layers.Dense(1))

optimizer = tf.keras.optimizers.Adam(0.01)

@tf.function
def train_step(inpt, targ):
  with tf.GradientTape() as g:
    model_out = model(inpt)
    model_loss = tf.reduce_mean(tf.square(tf.math.subtract(targ, model_out)))

  gradients = g.gradient(model_loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))
  return model_loss

train_ds = tf.data.Dataset.from_tensor_slices((inputs, targets))
train_ds = train_ds.repeat(2000).batch(200)

def train(train_ds):
  for inpt, targ in train_ds:
    model_loss = train_step(inpt, targ)
  tf.print(model_loss)

train(train_ds)
plot(tf.squeeze(model(inputs)))
</code></pre>

<p><a href="https://i.sstatic.net/GvgB5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GvgB5.png" alt="enter image description here"></a></p>

<python><tensorflow><keras><gradient-descent>
How to reduce variance of the model loss during training?
dereks
# How to reduce variance of the model loss during training?

(Asked by: username_0 on 2019-12-01)

I know that stochastic gradient descent always gives different results. What are the best practices to reduce this variance today?
I tried to predict simple function with two different approaches and every time I train them I see very different results.
Input data:
def plot(model_out):
  fig, ax = plt.subplots()
  ax.grid(True, which='both')
  ax.axhline(y=0, color='k', linewidth=1)
  ax.axvline(x=0, color='k', linewidth=1)

  ax.plot(x_line, y_line, c='g', linewidth=1)
  ax.scatter(inputs, targets, c='b', s=8)
  ax.scatter(inputs, model_out, c='r', s=8)

a = 5.0; b = 3.0; x_left, x_right = -16., 16.
NUM_EXAMPLES = 200
noise   = tf.random.normal((NUM_EXAMPLES,1))

inputs  = tf.random.uniform((NUM_EXAMPLES,1), x_left, x_right)
targets = a * tf.sin(inputs) + b + noise
x_line  = tf.linspace(x_left, x_right, 500)
y_line  = a * tf.sin(x_line) + b

Keras training:
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(50, activation='relu', input_shape=(1,)))
model.add(tf.keras.layers.Dense(50, activation='relu'))
model.add(tf.keras.layers.Dense(1))

model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam(0.01))
model.fit(inputs, targets, batch_size=200, epochs=2000, verbose=0)

print(model.evaluate(inputs, targets, verbose=0))
plot(model.predict(inputs))


Manual training:
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(50, activation='relu', input_shape=(1,)))
model.add(tf.keras.layers.Dense(50, activation='relu'))
model.add(tf.keras.layers.Dense(1))

optimizer = tf.keras.optimizers.Adam(0.01)

@tf.function
def train_step(inpt, targ):
  with tf.GradientTape() as g:
    model_out = model(inpt)
    model_loss = tf.reduce_mean(tf.square(tf.math.subtract(targ, model_out)))

  gradients = g.gradient(model_loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))
  return model_loss

train_ds = tf.data.Dataset.from_tensor_slices((inputs, targets))
train_ds = train_ds.repeat(2000).batch(200)

def train(train_ds):
  for inpt, targ in train_ds:
    model_loss = train_step(inpt, targ)
  tf.print(model_loss)

train(train_ds)
plot(tf.squeeze(model(inputs)))

**Tags:** python, tensorflow, keras, gradient-descent


---

## 1 answers

### Answer by username_1 on 2019-12-01. Score: 1

There are a few things you can play with:

* Try reducing the learning rate, or increasing decay.
* Try using regularization(L1/L2 or dropout)
* Try using momentum(your model may be stuck in a local minima)
* Adjust other hyperparams(nodes, layers, batch size, etc.)

Unless you have some knowledge about the specific cause of high loss variance, the above steps in some amount should get you where you need to go.

> **Comments:**
> **username_0**: How to reduce variance with fixed hyperparameters? That was the question.  

---

