Answers.Body: <p>By default, Keras sets shuffle argument <code>True</code>, so you should set the numpy seed before importing Keras.</p>
  <p><strong>CPU</strong></p>
  <pre><code>from numpy.random import seed
  seed(25)
  from keras.models import Sequential
  </code></pre>
  <p><strong>GPU</strong></p>
  <pre><code>tf.random.set_seed(seed)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T05:50:37.563Z
Answers.Text: Hi yahkyo, I set the random seed to a fixed value and rerun the whole model but the outcomes still changed every training. So I then set the shuffle = False, the outcomes were still not exactly the same but this time the trends were generally the same.
Answers.UserDisplayName: Leo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T05:58:53.473Z
Answers.Text: Hi Leo, I think random shuffle in your code is on GPU. However numpy works on CPU. Have you tried  `tf.random.set_seed(seed)` ?
Answers.UserDisplayName: yakhyo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T06:14:36.813Z
Answers.Text: You are right, I was running GPU. `tf.random.set_seed(seed)` this code doesn't give the 100% same outcomes, but I think  95% similarity is good for parameter adjustment. Thanks.
Answers.UserDisplayName: Leo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T05:17:32.227Z
Answers.LastActivityDate: 2021-01-26T15:02:15.867Z
Answers.LastEditDate: 2021-01-26T15:02:15.867Z
Answers.UserDisplayName: yakhyo
Body: <p>Here is a model that trains time series data in (batch, step, features) way.</p>
  <p>I have kept the random state for train test split function the same. Every parameter below the same, running the model training yields different outcomes every time and the outcomes are drastically different.</p>
  <p>What may be the factors that led to this? Regularization?</p>
  <p><a href="https://i.sstatic.net/rWl8m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rWl8m.png" alt="outcome_1" /></a>
  <a href="https://i.sstatic.net/rLM2N.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rLM2N.jpg" alt="outcome_2" /></a>
  <a href="https://i.sstatic.net/Cpmpt.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Cpmpt.jpg" alt="outcome_3" /></a></p>
  <p><code>X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=666)</code></p>
  <pre><code>def attention_model(X_train, y_train, X_test, y_test,num_classes,dropout=0.2, batch_size=68, learning_rate=0.0001,epochs=20,optimizer='Adam'):
      
      Dense_unit = 12
      LSTM_unit = 12
      
      attention_param = LSTM_unit*2
      attention_init_value = 1.0/attention_param
      
      
      u_train = np.full((X_train.shape[0], attention_param),
                        attention_init_value, dtype=np.float32)
      u_test = np.full((X_test.shape[0],attention_param),
                       attention_init_value, dtype=np.float32)
      
      
      with keras.backend.name_scope('BLSTMLayer'):
          # Bi-directional Long Short-Term Memory for learning the temporal aggregation
          input_feature = Input(shape=(X_train.shape[1],X_train.shape[2]))
          x = Masking(mask_value=0)(input_feature)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
          x = Dropout(dropout)(x)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
          x = Dropout(dropout)(x)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
          x = Dropout(dropout)(x)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
          x = Dropout(dropout)(x)
  
  
          y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.000029),kernel_regularizer=l2(0.027),recurrent_regularizer=l2(0.025),return_sequences=True, dropout=dropout))(x)
  #         y = Bidirectional(LSTM(LSTM_unit, kernel_regularizer=l2(0.01),recurrent_regularizer=l2(0.01), return_sequences=True, dropout=dropout))(y)
  
      with keras.backend.name_scope('AttentionLayer'):
          # Logistic regression for learning the attention parameters with a standalone feature as input
          input_attention = Input(shape=(LSTM_unit * 2,))
          u = Dense(LSTM_unit * 2, activation='softmax')(input_attention)
  
          # To compute the final weights for the frames which sum to unity
          alpha = dot([u, y], axes=-1)  # inner prod.
          alpha = Activation('softmax')(alpha)
  
      with keras.backend.name_scope('WeightedPooling'):
          # Weighted pooling to get the utterance-level representation
          z = dot([alpha, y], axes=1)
  
      # Get posterior probability for each emotional class
      output = Dense(num_classes, activation='softmax')(z)
  
      model = Model(inputs=[input_attention, input_feature], outputs=output)
  
      optimizer = opt_select(optimizer,learning_rate)
      model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=optimizer)
  
  
      hist = model.fit([u_train, X_train], 
                       y_train, 
                       batch_size=batch_size, 
                       epochs=epochs, 
                       verbose=2, 
                       validation_data=([u_test, X_test], y_test))
      
  
  #kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),
      return hist
  
  </code></pre>
  <pre><code>batch_size= 150
  #217
  epochs = 1000
  learning_rate = 0.00081
  optimizer = 'RMS'
  num_classes = y_train.shape[1]
  dropout=0.22
  
  tf.keras.backend.clear_session()
  
  history = attention_model(X_train, y_train, X_test, y_test, num_classes,dropout = dropout,batch_size=batch_size, learning_rate=learning_rate,epochs=epochs,optimizer=optimizer
  )
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-26T15:05:10.773Z
Comments.Text: Hello. Welcome to AI SE. I don't know if this is "just" a programming issue or not, but I would like you to note that programming questions (like "Why am I getting this programming error/bug?") are generally off-topic here. These questions are better suited for Stack Overflow. I don't know if your question is a pure programming issue (because I didn't read it, but it contains code...), but just keep this in mind. You should read https://ai.stackexchange.com/help/on-topic for more info.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-27T01:10:49.263Z
Comments.Text: @nbro Hi nbro, I was not sure what was causing the problem. My initial thought was that the problem was caused by some randomness of models not because of codes so I post it here.
Comments.UserDisplayName: Leo
Tags: <machine-learning><deep-learning><recurrent-neural-networks><attention>
Title: Factors that causing totally different outcomes from an exactly same model and datasets
UserDisplayName: Leo
ThreadText: # Factors that causing totally different outcomes from an exactly same model and datasets
  
  (Asked by: username_0 on 2021-01-26)
  
  Here is a model that trains time series data in (batch, step, features) way.
  I have kept the random state for train test split function the same. Every parameter below the same, running the model training yields different outcomes every time and the outcomes are drastically different.
  What may be the factors that led to this? Regularization?
  
  
  
  X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=666)
  def attention_model(X_train, y_train, X_test, y_test,num_classes,dropout=0.2, batch_size=68, learning_rate=0.0001,epochs=20,optimizer='Adam'):
      
      Dense_unit = 12
      LSTM_unit = 12
      
      attention_param = LSTM_unit*2
      attention_init_value = 1.0/attention_param
      
      
      u_train = np.full((X_train.shape[0], attention_param),
                        attention_init_value, dtype=np.float32)
      u_test = np.full((X_test.shape[0],attention_param),
                       attention_init_value, dtype=np.float32)
      
      
      with keras.backend.name_scope('BLSTMLayer'):
          # Bi-directional Long Short-Term Memory for learning the temporal aggregation
          input_feature = Input(shape=(X_train.shape[1],X_train.shape[2]))
          x = Masking(mask_value=0)(input_feature)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
          x = Dropout(dropout)(x)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
          x = Dropout(dropout)(x)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
          x = Dropout(dropout)(x)
          x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
          x = Dropout(dropout)(x)
  
  
          y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.000029),kernel_regularizer=l2(0.027),recurrent_regularizer=l2(0.025),return_sequences=True, dropout=dropout))(x)
  #         y = Bidirectional(LSTM(LSTM_unit, kernel_regularizer=l2(0.01),recurrent_regularizer=l2(0.01), return_sequences=True, dropout=dropout))(y)
  
      with keras.backend.name_scope('AttentionLayer'):
          # Logistic regression for learning the attention parameters with a standalone feature as input
          input_attention = Input(shape=(LSTM_unit * 2,))
          u = Dense(LSTM_unit * 2, activation='softmax')(input_attention)
  
          # To compute the final weights for the frames which sum to unity
          alpha = dot([u, y], axes=-1)  # inner prod.
          alpha = Activation('softmax')(alpha)
  
      with keras.backend.name_scope('WeightedPooling'):
          # Weighted pooling to get the utterance-level representation
          z = dot([alpha, y], axes=1)
  
      # Get posterior probability for each emotional class
      output = Dense(num_classes, activation='softmax')(z)
  
      model = Model(inputs=[input_attention, input_feature], outputs=output)
  
      optimizer = opt_select(optimizer,learning_rate)
      model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=optimizer)
  
  
      hist = model.fit([u_train, X_train], 
                       y_train, 
                       batch_size=batch_size, 
                       epochs=epochs, 
                       verbose=2, 
                       validation_data=([u_test, X_test], y_test))
      
  
  #kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),
      return hist
  
  
  batch_size= 150
  #217
  epochs = 1000
  learning_rate = 0.00081
  optimizer = 'RMS'
  num_classes = y_train.shape[1]
  dropout=0.22
  
  tf.keras.backend.clear_session()
  
  history = attention_model(X_train, y_train, X_test, y_test, num_classes,dropout = dropout,batch_size=batch_size, learning_rate=learning_rate,epochs=epochs,optimizer=optimizer
  )
  
  **Tags:** machine-learning, deep-learning, recurrent-neural-networks, attention
  > **Comments:**
  > **username_1**: Hello. Welcome to AI SE. I don't know if this is "just" a programming issue or not, but I would like you to note that programming questions (like "Why am I getting this programming error/bug?") are generally off-topic here. These questions are better suited for Stack Overflow. I don't know if your question is a pure programming issue (because I didn't read it, but it contains code...), but just keep this in mind. You should read https://ai.stackexchange.com/help/on-topic for more info.  
  > **username_0**: @nbro Hi nbro, I was not sure what was causing the problem. My initial thought was that the problem was caused by some randomness of models not because of codes so I post it here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-26. Score: 0 | Accepted answer
  
  By default, Keras sets shuffle argument True, so you should set the numpy seed before importing Keras.
  CPU
  from numpy.random import seed
  seed(25)
  from keras.models import Sequential
  
  GPU
  tf.random.set_seed(seed)
  
  > **Comments:**
  > **username_0**: Hi yahkyo, I set the random seed to a fixed value and rerun the whole model but the outcomes still changed every training. So I then set the shuffle = False, the outcomes were still not exactly the same but this time the trends were generally the same.  
  > **username_2**: Hi Leo, I think random shuffle in your code is on GPU. However numpy works on CPU. Have you tried  `tf.random.set_seed(seed)` ?  
  > **username_0**: You are right, I was running GPU. `tf.random.set_seed(seed)` this code doesn't give the 100% same outcomes, but I think  95% similarity is good for parameter adjustment. Thanks.  
  
  ---
  
  