Answers.Body: <p>From comments, you are testing your ideas with a very small neural networks.</p>
  <p>The highly variable end result with large dependency on initial conditions is a common result of working with small numbers of neurons. Random initialisation works in part because you can reasonably expect a mean activation value when considered over a layer's output. With a very small number of neurons in any layer, this is less likely to happen. The assumptions behind the statistical analysis that suggest how to properly initialise the parameters in a layer don't hold for very small layers.</p>
  <p>Increasing <code>n_units_1</code> and <code>n_units_2</code> to a slightly higher value, say <code>10</code>, should resolve this provided you are using a normal initialisation routine, such as Keras' default.</p>
  <p>Another thing that can help is adding some regularisation other than early stopping. A small amount of weight decay (or L2 weight normalisation) may work. Usually you would not apply regularisation to very low number of neurons though, since the low learning capacity of the network means it is already quite restricted. However, one of the issues you may face when increasing the size of the network is that its learning capacity will be much larger than the simple problem you currently are testing it with. So you <em>may</em> need to start addressing over-fitting. Try without it initially though.</p>
  <p>There is probably a sweet spot for the layer sizes, where 99%+ of your runs will be stable and accurate over the whole domain, but you won't need any more regularisation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-17T08:01:39.620Z
Answers.LastActivityDate: 2023-11-17T08:13:51.630Z
Answers.LastEditDate: 2023-11-17T08:13:51.630Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am training a simple RNN model in <code>keras</code> to predict a time series.</p>
  <p>The time series I am considering is just a sine function</p>
  <p><a href="https://i.sstatic.net/iQUv7.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iQUv7.jpg" alt="enter image description here" /></a></p>
  <p>The task to solve is the following:</p>
  <blockquote>
  <p>Given a timeseries of 90 time steps, predict the next 10 time steps.</p>
  </blockquote>
  <p>I am generating my training set by sampling from</p>
  <p><a href="https://i.sstatic.net/VC0xY.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VC0xY.jpg" alt="enter image description here" /></a></p>
  <p>where <span class="math-container">$\epsilon$</span> is just a noise term that shifts the series. I have added the noise to have a good training sample.</p>
  <p><em><strong>Question</strong></em></p>
  <p>I train my model for 10 epochs. I notice that sometimes the predictions are actually very good. Some other times, if I retrain the same model with the same dataset, the predictions are very bad. This has to do with the randomness of weights initialisations, but I was hoping that the training algorithm would have found the &quot;best&quot; solution anyway, irrespectively of the initial weights. When I say &quot;best&quot; solution, I mean a class of solutions whose variance is small.</p>
  <p><strong>How can I deal with random weights initialisation? Do I need to add some regulation terms to my network?</strong></p>
  <p>I know that I can define an ensemble of multiple models in order to cope with weights initialisation. However, this is a very simple problem, and I was expecting to be able to solve it with a single simple model.</p>
  <p><em><strong>Description of the model and code</strong></em></p>
  <p>You can find a Jupyter notebook <a href="https://github.com/doogersPy/files/blob/main/dependence_on_initial_weights.ipynb" rel="nofollow noreferrer">here</a> .</p>
  <p>My training sample will be time series of lenght 90 and the targets will be the value of the series at the 91st step.</p>
  <p>I will predict the 10 steps by first predicting the 91st step. Then, I will plug it in the sample. I will select the last 90 terms and predict the 92nd step and so on.</p>
  <p>I generate the samples with this code</p>
  <pre><code>def generate_series(n_series, n_timesteps, scale_noise=0, scale_shift_origin=10):
      &quot;&quot;&quot;
      Generate random sin series for testing purposes.
      &quot;&quot;&quot;
      series = []
      for _ in range(n_series):
          noise = np.random.normal(loc=0.0, scale=scale_noise, size=n_timesteps)
          noise_origin = np.random.normal(loc=1, scale=scale_shift_origin, size=1)
          x = np.linspace(start=0, stop=100, num=n_timesteps)
          y = np.sin(0.3*(x + noise_origin)) + noise
          series.append(y)
      return np.array(series)
  
  keras.utils.set_random_seed(42)
  
  series = generate_series(10000,100)
  
  X_train, y_train = series[:8000,:-10], series[:8000,-10]
  X_train = X_train.reshape(8000, -1,1)
  y_train = y_train.reshape(8000, -1)
  
  X_test, y_test = series[8000:,:-10], series[8000:,-10]
  X_test = X_test.reshape(2000, -1,1)
  y_test = y_test.reshape(2000, -1)
  </code></pre>
  <p>The model I am using is very simple, it is a stack of two RNN layers and a Dense layer.</p>
  <pre><code>class TwoLSTM:
      def __init__(self, n_units_1:int, n_units_2:int):
          self.n_units_1 = n_units_1
          self.n_units_2 = n_units_2
          self.model = None
          
          self.build()
          
      def build(self):
          self.model = keras.models.Sequential([
          keras.layers.LSTM(self.n_units_1, return_sequences=True, input_shape=[None, 1]),
          keras.layers.LSTM(self.n_units_2),
          keras.layers.Dense(1)])
          
          self.model.compile(loss=&quot;mse&quot;, optimizer=&quot;adam&quot;)
      
      def fit(self, train:tuple, test:tuple, epochs:int, batch_size:int=32,verbose=0):
          X_train, y_train = train
          X_test, y_test = test
          
          callbacks = [
          keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, mode='min', restore_best_weights=True)]
          history = self.model.fit(X_train, y_train, 
                          epochs=epochs, batch_size=batch_size, 
                          validation_data=(X_test, y_test), callbacks=callbacks,
                          shuffle=True, verbose=verbose)
          return history
  </code></pre>
  <p>I generate the predictions with this code</p>
  <pre><code>def generate_predictions(series, model):
      X = series.reshape(1,-1,1)
      X_new = X[:,:90,:]
      for step in range(10):
          X_to_use = X_new[:,step:90+step,:]
          y_pred = model.predict(X_to_use, verbose=0).reshape(1,-1,1)
          X_new = np.concatenate([X_new, y_pred], axis=1)
      
      return X_new
  </code></pre>
  <p>The first training of 10 epochs returns the following prediction of a series in the training set</p>
  <p><a href="https://i.sstatic.net/m3HQV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m3HQV.png" alt="enter image description here" /></a></p>
  <p>and the log of the loss function during training are</p>
  <p><a href="https://i.sstatic.net/8eS0X.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8eS0X.png" alt="enter image description here" /></a></p>
  <p>Then, I train again the same model (by redefining it again; I don't use the best weights of the first run as starting point) and this time I get very good predictions and loss functions</p>
  <p><a href="https://i.sstatic.net/0jqXN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0jqXN.png" alt="enter image description here" /></a></p>
  <p>and</p>
  <p><a href="https://i.sstatic.net/Ipr88.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ipr88.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-16T23:16:43.383Z
Comments.Text: What values are you using for `n_units_1` and `n_units_2` for those graphs?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-17T06:47:55.710Z
Comments.Text: @NeilSlater sorry, I should have written it in the post. I am using `n_units_1` = `n_units_2` = 1. It is on the jupyter notebook though.
Comments.UserDisplayName: apt45
Tags: <neural-networks><recurrent-neural-networks><weights-initialization>
Title: How can I deal with random weights initialisation when predicting a time-series sine function?
UserDisplayName: apt45
ThreadText: # How can I deal with random weights initialisation when predicting a time-series sine function?
  
  (Asked by: username_0 on 2023-11-16)
  
  I am training a simple RNN model in `keras` to predict a time series.
  
  The time series I am considering is just a sine function
  
  [![enter image description here](https://i.sstatic.net/iQUv7.jpg)](https://i.sstatic.net/iQUv7.jpg)
  
  The task to solve is the following:
  
  > Given a timeseries of 90 time steps, predict the next 10 time steps.
  
  I am generating my training set by sampling from
  
  [![enter image description here](https://i.sstatic.net/VC0xY.jpg)](https://i.sstatic.net/VC0xY.jpg)
  
  where $\epsilon$ is just a noise term that shifts the series. I have added the noise to have a good training sample.
  
  ***Question***
  
  I train my model for 10 epochs. I notice that sometimes the predictions are actually very good. Some other times, if I retrain the same model with the same dataset, the predictions are very bad. This has to do with the randomness of weights initialisations, but I was hoping that the training algorithm would have found the "best" solution anyway, irrespectively of the initial weights. When I say "best" solution, I mean a class of solutions whose variance is small.
  
  **How can I deal with random weights initialisation? Do I need to add some regulation terms to my network?**
  
  I know that I can define an ensemble of multiple models in order to cope with weights initialisation. However, this is a very simple problem, and I was expecting to be able to solve it with a single simple model.
  
  ***Description of the model and code***
  
  You can find a Jupyter notebook [here](https://github.com/doogersPy/files/blob/main/dependence_on_initial_weights.ipynb) .
  
  My training sample will be time series of lenght 90 and the targets will be the value of the series at the 91st step.
  
  I will predict the 10 steps by first predicting the 91st step. Then, I will plug it in the sample. I will select the last 90 terms and predict the 92nd step and so on.
  
  I generate the samples with this code
  
  ```
  def generate_series(n_series, n_timesteps, scale_noise=0, scale_shift_origin=10):
      """
      Generate random sin series for testing purposes.
      """
      series = []
      for _ in range(n_series):
          noise = np.random.normal(loc=0.0, scale=scale_noise, size=n_timesteps)
          noise_origin = np.random.normal(loc=1, scale=scale_shift_origin, size=1)
          x = np.linspace(start=0, stop=100, num=n_timesteps)
          y = np.sin(0.3*(x + noise_origin)) + noise
          series.append(y)
      return np.array(series)
  
  keras.utils.set_random_seed(42)
  
  series = generate_series(10000,100)
  
  X_train, y_train = series[:8000,:-10], series[:8000,-10]
  X_train = X_train.reshape(8000, -1,1)
  y_train = y_train.reshape(8000, -1)
  
  X_test, y_test = series[8000:,:-10], series[8000:,-10]
  X_test = X_test.reshape(2000, -1,1)
  y_test = y_test.reshape(2000, -1)
  
  ```
  
  The model I am using is very simple, it is a stack of two RNN layers and a Dense layer.
  
  ```
  class TwoLSTM:
      def __init__(self, n_units_1:int, n_units_2:int):
          self.n_units_1 = n_units_1
          self.n_units_2 = n_units_2
          self.model = None
          
          self.build()
          
      def build(self):
          self.model = keras.models.Sequential([
          keras.layers.LSTM(self.n_units_1, return_sequences=True, input_shape=[None, 1]),
          keras.layers.LSTM(self.n_units_2),
          keras.layers.Dense(1)])
          
          self.model.compile(loss="mse", optimizer="adam")
      
      def fit(self, train:tuple, test:tuple, epochs:int, batch_size:int=32,verbose=0):
          X_train, y_train = train
          X_test, y_test = test
          
          callbacks = [
          keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, mode='min', restore_best_weights=True)]
          history = self.model.fit(X_train, y_train, 
                          epochs=epochs, batch_size=batch_size, 
                          validation_data=(X_test, y_test), callbacks=callbacks,
                          shuffle=True, verbose=verbose)
          return history
  
  ```
  
  I generate the predictions with this code
  
  ```
  def generate_predictions(series, model):
      X = series.reshape(1,-1,1)
      X_new = X[:,:90,:]
      for step in range(10):
          X_to_use = X_new[:,step:90+step,:]
          y_pred = model.predict(X_to_use, verbose=0).reshape(1,-1,1)
          X_new = np.concatenate([X_new, y_pred], axis=1)
      
      return X_new
  
  ```
  
  The first training of 10 epochs returns the following prediction of a series in the training set
  
  [![enter image description here](https://i.sstatic.net/m3HQV.png)](https://i.sstatic.net/m3HQV.png)
  
  and the log of the loss function during training are
  
  [![enter image description here](https://i.sstatic.net/8eS0X.png)](https://i.sstatic.net/8eS0X.png)
  
  Then, I train again the same model (by redefining it again; I don't use the best weights of the first run as starting point) and this time I get very good predictions and loss functions
  
  [![enter image description here](https://i.sstatic.net/0jqXN.png)](https://i.sstatic.net/0jqXN.png)
  
  and
  
  [![enter image description here](https://i.sstatic.net/Ipr88.png)](https://i.sstatic.net/Ipr88.png)
  
  **Tags:** neural-networks, recurrent-neural-networks, weights-initialization
  
  > **Comments:**
  > **username_1**: What values are you using for `n_units_1` and `n_units_2` for those graphs?  
  > **username_0**: @username_1 sorry, I should have written it in the post. I am using `n_units_1` = `n_units_2` = 1. It is on the jupyter notebook though.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-17. Score: 1
  
  From comments, you are testing your ideas with a very small neural networks.
  
  The highly variable end result with large dependency on initial conditions is a common result of working with small numbers of neurons. Random initialisation works in part because you can reasonably expect a mean activation value when considered over a layer's output. With a very small number of neurons in any layer, this is less likely to happen. The assumptions behind the statistical analysis that suggest how to properly initialise the parameters in a layer don't hold for very small layers.
  
  Increasing `n_units_1` and `n_units_2` to a slightly higher value, say `10`, should resolve this provided you are using a normal initialisation routine, such as Keras' default.
  
  Another thing that can help is adding some regularisation other than early stopping. A small amount of weight decay (or L2 weight normalisation) may work. Usually you would not apply regularisation to very low number of neurons though, since the low learning capacity of the network means it is already quite restricted. However, one of the issues you may face when increasing the size of the network is that its learning capacity will be much larger than the simple problem you currently are testing it with. So you *may* need to start addressing over-fitting. Try without it initially though.
  
  There is probably a sweet spot for the layer sizes, where 99%+ of your runs will be stable and accurate over the whole domain, but you won't need any more regularisation.
  
  
  ---
  
  