Answers.Body: <p>Sorry cannot directly reply to your comment as I posted without an account, and you were right! I replaced transposed layers with Upscale1D+Conv1D and that solved the issue.</p>
  <pre><code>gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
  </code></pre>
  <p>should become (notice that strides=2 becomes strides=1):</p>
  <pre><code>gen = Upscale1D()(gen)
  gen = Conv1D(128, 4, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-19T15:41:25.147Z
Answers.Text: I am glad that worked. If you have time to write a self-answer here with abit more detail, I will definitely upvote it. For instance if you described the working architecture in this answer with the same level of detail as in the question
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-19T15:50:00.800Z
Answers.Text: Please, follow the steps [here](https://ai.stackexchange.com/help/merging-accounts) in order to merge your just created account with the unregistered one.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-19T15:38:31.703Z
Answers.LastActivityDate: 2021-02-19T15:46:19.663Z
Answers.LastEditDate: 2021-02-19T15:46:19.663Z
Answers.UserDisplayName: Hazar
Body: <p>I am training a Semi-Supervised GAN, using multivariate time-series with window of shape (180*80) with the generator and discriminator architecture below. My data is scaled using Robust Scaler, so I kept linear activation for the generator output.</p>
  <p>During the training I get noise in the generated signals and I can't understand the reason why whereas the original data is smooth. What can be the reason for this noise?</p>
  <p><a href="https://i.sstatic.net/2TpKY.png" rel="noreferrer"><img src="https://i.sstatic.net/2TpKY.png" alt="A generated signal." /></a><a href="https://i.sstatic.net/amY04.png" rel="noreferrer"><img src="https://i.sstatic.net/amY04.png" alt="Another generated signal." /></a></p>
  <pre><code>def make_generator_model(noise):
      w_init = tf.random_normal_initializer(stddev=0.02)
      gamma_init = tf.random_normal_initializer(1., 0.02)
      
      def residual_layer(layer_input):
          
          res_block = Conv1D(128, 3, strides=1, padding='same')(layer_input)
          res_block = BatchNormalization(gamma_initializer=gamma_init)(res_block)
          res_block = LeakyReLU()(res_block)
          res_block = Conv1D(128, 3, strides=1, padding='same')(res_block)
          res_block = BatchNormalization(gamma_initializer=gamma_init)(res_block)
          res_block = LeakyReLU()(res_block)
          res_add = Add()([res_block, layer_input])
          
          return res_add
      
      in_noise = Input(shape=(100,))
      
  
      gen = Dense(180*65, kernel_initializer=w_init, use_bias=None)(in_noise)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
  
      gen = Reshape((180, 65))(gen)
      #assert model.output_shape == (None, 45, 256) # Note: None is the batch size
  
      gen = Conv1D(64, 7, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 45, 128)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
          
      gen = Conv1D(64, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 45, 128)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
      gen = Conv1D(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 45, 128)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
      for i in range(6):
          gen = residual_layer(gen)
  
      gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 90, 64)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
      gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 90, 64)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
  
      out_layer = Conv1D(65, 7, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 180, 65)
      
      model = Model(in_noise, out_layer)
  
      return model
  
  def make_discriminator_model(n_classes=8):
      w_init = tf.random_normal_initializer(stddev=0.02)
      gamma_init = tf.random_normal_initializer(1., 0.02)   
      
      in_window = Input(shape=(180, 65))
  
      disc = Conv1D(64, 4, strides=1, padding='same', kernel_initializer=w_init)(in_window)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
  
      disc = Conv1D(64*2, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Conv1D(64*4, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Conv1D(64*8, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Conv1D(64*16, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Flatten()(disc)
      
      disc = Dense(128)(disc)
      disc = Dense(128)(disc)
      
      out_layer = Dense(1)(disc)
      
      c_out_layer = Dense(8, activation='softmax')(disc)
      
      model = Model(in_window, out_layer)
      c_model = Model(in_window, c_out_layer)
  
      return model, c_model
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-19T14:17:11.797Z
Comments.Text: I think at least some of the noise may be because you are using `Conv1DTranspose` to scale up directly. You might prefer to scale up using simple interpolation then run a normal convolution instead. Hopefully someone can provide a proper answer to this, but intuitively it is related to image GANs' fondness for grid patterns, which is similar high frequency noise.
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><convolutional-neural-networks><generative-adversarial-networks><semi-supervised-learning>
Title: GAN Generator Output w/ Periodic Noise
UserDisplayName: Hazar
ThreadText: # GAN Generator Output w/ Periodic Noise
  
  (Asked by: username_0 on 2021-02-19)
  
  I am training a Semi-Supervised GAN, using multivariate time-series with window of shape (180\*80) with the generator and discriminator architecture below. My data is scaled using Robust Scaler, so I kept linear activation for the generator output.
  
  During the training I get noise in the generated signals and I can't understand the reason why whereas the original data is smooth. What can be the reason for this noise?
  
  [![A generated signal.](https://i.sstatic.net/2TpKY.png)](https://i.sstatic.net/2TpKY.png)[![Another generated signal.](https://i.sstatic.net/amY04.png)](https://i.sstatic.net/amY04.png)
  
  ```
  def make_generator_model(noise):
      w_init = tf.random_normal_initializer(stddev=0.02)
      gamma_init = tf.random_normal_initializer(1., 0.02)
      
      def residual_layer(layer_input):
          
          res_block = Conv1D(128, 3, strides=1, padding='same')(layer_input)
          res_block = BatchNormalization(gamma_initializer=gamma_init)(res_block)
          res_block = LeakyReLU()(res_block)
          res_block = Conv1D(128, 3, strides=1, padding='same')(res_block)
          res_block = BatchNormalization(gamma_initializer=gamma_init)(res_block)
          res_block = LeakyReLU()(res_block)
          res_add = Add()([res_block, layer_input])
          
          return res_add
      
      in_noise = Input(shape=(100,))
      
  
      gen = Dense(180*65, kernel_initializer=w_init, use_bias=None)(in_noise)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
  
      gen = Reshape((180, 65))(gen)
      #assert model.output_shape == (None, 45, 256) # Note: None is the batch size
  
      gen = Conv1D(64, 7, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 45, 128)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
          
      gen = Conv1D(64, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 45, 128)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
      gen = Conv1D(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 45, 128)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
      for i in range(6):
          gen = residual_layer(gen)
  
      gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 90, 64)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
      gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 90, 64)
      gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
      gen = LeakyReLU()(gen)
      
  
      out_layer = Conv1D(65, 7, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
      #assert model.output_shape == (None, 180, 65)
      
      model = Model(in_noise, out_layer)
  
      return model
  
  def make_discriminator_model(n_classes=8):
      w_init = tf.random_normal_initializer(stddev=0.02)
      gamma_init = tf.random_normal_initializer(1., 0.02)   
      
      in_window = Input(shape=(180, 65))
  
      disc = Conv1D(64, 4, strides=1, padding='same', kernel_initializer=w_init)(in_window)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
  
      disc = Conv1D(64*2, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Conv1D(64*4, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Conv1D(64*8, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Conv1D(64*16, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
      disc = LeakyReLU()(disc)
      disc = Dropout(0.3)(disc)
      
      disc = Flatten()(disc)
      
      disc = Dense(128)(disc)
      disc = Dense(128)(disc)
      
      out_layer = Dense(1)(disc)
      
      c_out_layer = Dense(8, activation='softmax')(disc)
      
      model = Model(in_window, out_layer)
      c_model = Model(in_window, c_out_layer)
  
      return model, c_model
  
  ```
  
  **Tags:** neural-networks, convolutional-neural-networks, generative-adversarial-networks, semi-supervised-learning
  
  > **Comments:**
  > **username_1**: I think at least some of the noise may be because you are using `Conv1DTranspose` to scale up directly. You might prefer to scale up using simple interpolation then run a normal convolution instead. Hopefully someone can provide a proper answer to this, but intuitively it is related to image GANs' fondness for grid patterns, which is similar high frequency noise.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-02-19. Score: 4
  
  Sorry cannot directly reply to your comment as I posted without an account, and you were right! I replaced transposed layers with Upscale1D+Conv1D and that solved the issue.
  
  ```
  gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
  
  ```
  
  should become (notice that strides=2 becomes strides=1):
  
  ```
  gen = Upscale1D()(gen)
  gen = Conv1D(128, 4, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
  
  ```
  
  > **Comments:**
  > **username_1**: I am glad that worked. If you have time to write a self-answer here with abit more detail, I will definitely upvote it. For instance if you described the working architecture in this answer with the same level of detail as in the question  
  > **username_2**: Please, follow the steps [here](https://ai.stackexchange.com/help/merging-accounts) in order to merge your just created account with the unregistered one.  
  
  ---
  
  