Answers.Body: <p><em>Is it possible that the discriminator picks on this phenomenon?</em></p>
  <p>It's not just possible, it's a certainty. The generator should learn to translate an input distribution A to an output distribution B, if distribution B has range <span class="math-container">$(-\infty, \infty)$</span> and your generator can output only values in range <span class="math-container">$(-1, 1)$</span> the translation between the two simply can't happen.</p>
  <p>Move to another normalization, like min max, and replace tanh with sigmoid as final activation function. If your GAN will still have trouble converging, only then you might start investigating other components like the discriminator depth.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T07:55:13.320Z
Answers.Text: is there a way to do the min max normalization easily? I've checked https://stackoverflow.com/questions/68791508/min-max-normalization-of-a-tensor-in-pytorch but it looks complicated. Also is there not a final activation function that looks like tanh (will likely outputs between -1 and 1) but can output values in (−∞,∞)?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T08:02:08.407Z
Answers.Text: you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T08:05:23.283Z
Answers.Text: the min max normalization is actually probably the simplest one among normalizations. You just subtract the minimum value of the whole image to all pixels, and divide them by (max value of whole image - min value of whole image). The first step push down the min value to zero, the second step push down the max value to 1, so your guarantee to have values within the range $(0,1)$ (hence the necessity of using sigmoid activation with this normalization)
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T09:22:47.227Z
Answers.Text: @EdoardoGuerriero do you have anything where you base this on (that you advice against linear activations due to stability issues)? I'm being curious, because I've had great results with using linear output activations where my output distribution was not constrained.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T09:42:04.017Z
Answers.Text: @Edoardo Guerriero ok! Actually, I just remembered, but aren't the values of the MNIST dataset already between 0 and 1 anyway? Why do people use Normalize in that case (https://github.com/lyeoni/pytorch-mnist-GAN, etc.)
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T10:04:19.997Z
Answers.Text: @FluidMechanicsPotentialFlows I guess it's due to a not really good understanding of data preprocessing, combined with the fact many tutorials online use tanh as final activation. But testing both configuration is a quick exercise for only 2 line of code to modify.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T10:05:57.443Z
Answers.Text: @EdoardoGuerriero Yeh of course, I just wanted to know why people were doing it in general. I'm gonna try to not use any activation at all and to use a sigmoid with a min max normalization (but I think that's already been done in the Dataset I'm  using, I'd have to check)
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T10:08:56.460Z
Answers.Text: @Chillston I got that experience while working on satellite data super resolution. Admittedly the data where codified in uint16, so the range to learn is quite huge compared to uint8. We tried to not normalized cause we didn't wanted to affect the raw sensor data but we gave up pretty quickly on this potential risk and after normalizing the training was much more easy. Still, the mnist data is quite a simple one so you might be right that this risk is probably not noticeable.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T17:48:12.237Z
Answers.Text: Ah, I was asking about the output activation. You where advising against linear activation, because it can lead to unstable training (I was asking about this comment: "you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable."). This is not clear to me, I had good experience with linear output activation so far, but I am very interested to learn more about this.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T07:15:16.413Z
Answers.LastActivityDate: 2022-03-29T07:15:16.413Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm currently coding a GAN on the dataset MNIST.
  I'm using the following code to transform my data:</p>
  <pre><code># MNIST Dataset
  transform = transforms.Compose([
      transforms.ToTensor(),
      transforms.Normalize((0.1307,), (0.3081,))])
  # the output of torchvision datasets are PILImage images of range [0, 1] and we 
  # want data that is centered around 0 with a std of 1 (0.1307 and 0.3081 are the estimated values of the MNIST mean &amp; std)
  </code></pre>
  <p>I will have data centered around 0 with a standard deviation of 1 ((0.1307,), (0.3081,) are the estimated mean &amp; standard deviation on the training dataset)). <strong>So that means that there will occasionnaly be values above 1 and below -1 in the real data.</strong></p>
  <p>Now, my generator ends up with a tanh activation function:</p>
  <pre><code>return torch.tanh(self.fc4(x))  # outputs in[-1; 1]
  </code></pre>
  <p><strong>That means there will never be values above 1 and below -1 in the faked data</strong>.</p>
  <p>Is it possible that the discriminator picks on this phenomenon? This seems to be the case as its loss goes to 0 really quickly. However this also could also be the case that the discriminator is just &quot;too strong&quot; as I've seen numerous times on stackexchange posts. I however never seen nobody talking about the fact that it could pick on the fact that there are outliers in the &quot;real data&quot; and only pixels between -1 and 1 in the &quot;fake data&quot;.</p>
  <p>EDIT: the entirety of my code can be found here:
  <a href="https://github.com/JQuentinMendoza2008/PyTorch_GAN_for_MNIST_Dataset" rel="nofollow noreferrer">https://github.com/JQuentinMendoza2008/PyTorch_GAN_for_MNIST_Dataset</a></p>
  <p>Any suggestion is welcomed.</p>
  
Tags: <generative-adversarial-networks><pytorch>
Title: GANs inputs normalized and generator only outputs in [-1; 1]
UserDisplayName: FluidMechanics Potential Flows
ThreadText: # GANs inputs normalized and generator only outputs in [-1; 1]
  
  (Asked by: username_0 on 2022-03-28)
  
  I'm currently coding a GAN on the dataset MNIST.
  I'm using the following code to transform my data:
  # MNIST Dataset
  transform = transforms.Compose([
      transforms.ToTensor(),
      transforms.Normalize((0.1307,), (0.3081,))])
  # the output of torchvision datasets are PILImage images of range [0, 1] and we 
  # want data that is centered around 0 with a std of 1 (0.1307 and 0.3081 are the estimated values of the MNIST mean & std)
  
  I will have data centered around 0 with a standard deviation of 1 ((0.1307,), (0.3081,) are the estimated mean & standard deviation on the training dataset)). So that means that there will occasionnaly be values above 1 and below -1 in the real data.
  Now, my generator ends up with a tanh activation function:
  return torch.tanh(self.fc4(x))  # outputs in[-1; 1]
  
  That means there will never be values above 1 and below -1 in the faked data.
  Is it possible that the discriminator picks on this phenomenon? This seems to be the case as its loss goes to 0 really quickly. However this also could also be the case that the discriminator is just "too strong" as I've seen numerous times on stackexchange posts. I however never seen nobody talking about the fact that it could pick on the fact that there are outliers in the "real data" and only pixels between -1 and 1 in the "fake data".
  EDIT: the entirety of my code can be found here:
  https://github.com/JQuentinMendoza2008/PyTorch_GAN_for_MNIST_Dataset
  Any suggestion is welcomed.
  
  **Tags:** generative-adversarial-networks, pytorch
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-29. Score: 1
  
  *Is it possible that the discriminator picks on this phenomenon?*
  
  It's not just possible, it's a certainty. The generator should learn to translate an input distribution A to an output distribution B, if distribution B has range $(-\infty, \infty)$ and your generator can output only values in range $(-1, 1)$ the translation between the two simply can't happen.
  
  Move to another normalization, like min max, and replace tanh with sigmoid as final activation function. If your GAN will still have trouble converging, only then you might start investigating other components like the discriminator depth.
  
  > **Comments:**
  > **username_0**: is there a way to do the min max normalization easily? I've checked https://stackoverflow.com/questions/68791508/min-max-normalization-of-a-tensor-in-pytorch but it looks complicated. Also is there not a final activation function that looks like tanh (will likely outputs between -1 and 1) but can output values in (−∞,∞)?  
  > **username_1**: you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable.  
  > **username_1**: the min max normalization is actually probably the simplest one among normalizations. You just subtract the minimum value of the whole image to all pixels, and divide them by (max value of whole image - min value of whole image). The first step push down the min value to zero, the second step push down the max value to 1, so your guarantee to have values within the range $(0,1)$ (hence the necessity of using sigmoid activation with this normalization)  
  > **username_2**: @EdoardoGuerriero do you have anything where you base this on (that you advice against linear activations due to stability issues)? I'm being curious, because I've had great results with using linear output activations where my output distribution was not constrained.  
  > **username_0**: @Edoardo Guerriero ok! Actually, I just remembered, but aren't the values of the MNIST dataset already between 0 and 1 anyway? Why do people use Normalize in that case (https://github.com/lyeoni/pytorch-mnist-GAN, etc.)  
  > **username_1**: @FluidMechanicsPotentialFlows I guess it's due to a not really good understanding of data preprocessing, combined with the fact many tutorials online use tanh as final activation. But testing both configuration is a quick exercise for only 2 line of code to modify.  
  > **username_0**: @EdoardoGuerriero Yeh of course, I just wanted to know why people were doing it in general. I'm gonna try to not use any activation at all and to use a sigmoid with a min max normalization (but I think that's already been done in the Dataset I'm  using, I'd have to check)  
  > **username_1**: @Chillston I got that experience while working on satellite data super resolution. Admittedly the data where codified in uint16, so the range to learn is quite huge compared to uint8. We tried to not normalized cause we didn't wanted to affect the raw sensor data but we gave up pretty quickly on this potential risk and after normalizing the training was much more easy. Still, the mnist data is quite a simple one so you might be right that this risk is probably not noticeable.  
  > **username_2**: Ah, I was asking about the output activation. You where advising against linear activation, because it can lead to unstable training (I was asking about this comment: "you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable."). This is not clear to me, I had good experience with linear output activation so far, but I am very interested to learn more about this.  
  
  ---
  
  