Answers.Body: <p>The sudden appearance of NaN values in the loss function after introducing chromatically adapted images suggests that the distribution of training data has changed in a way that causes numerical instabilities during gradient updates. Consider a generator <span class="math-container">$G$</span> (for inpainting) and a discriminator <span class="math-container">$D$</span> (for distinguishing real from generated samples). A typical generator objective is</p>
  <p><span class="math-container">$$L_{\mathrm{gen}}(G) = \alpha\,L_{\mathrm{rec}}(G) + \beta\,L_{\mathrm{adv}}(G),$$</span></p>
  <p>where <span class="math-container">$L_{\mathrm{rec}}(G)$</span> might be an <span class="math-container">$L_1$</span> inpainting loss of the form
  <span class="math-container">$$
  L_{\mathrm{rec}}(G) = \frac{1}{N}\sum_{i=1}^N \bigl\lVert (G(x_i,m_i) - x_i)\odot (1-m_i)\bigr\rVert_1,
  $$</span>
  and <span class="math-container">$L_{\mathrm{adv}}(G)$</span> can be
  <span class="math-container">$$
  L_{\mathrm{adv}}(G) = -\frac{1}{N}\sum_{i=1}^N \log\bigl(D(x_i',\,G(x_i,m_i))\bigr),
  $$</span>
  where <span class="math-container">$x_i$</span> is the real image, <span class="math-container">$m_i$</span> is the mask, and <span class="math-container">$x_i'$</span> might be a ground truth version of <span class="math-container">$x_i$</span>. The discriminator objective <span class="math-container">$L_{\mathrm{disc}}$</span> often takes the form
  <span class="math-container">$$
  L_{\mathrm{disc}}(D) = -\frac{1}{N}\sum_{i=1}^N \Bigl[\log\bigl(D(x_i',\,x_i)\bigr) + \log\bigl(1 - D(x_i',\,G(x_i,m_i)))\Bigr].
  $$</span></p>
  <p>If the new images shift color statistics sufficiently, the discriminator may produce unbounded gradients that cause floating-point overflows in weight updates. A smaller learning rate delays but does not prevent this if <span class="math-container">$\|\nabla_w L_{\mathrm{gen}}\|$</span> can still grow large. Techniques that mitigate the domain shift and dampen exploding gradients can stabilize training, such as gradient penalties (like a Lipschitz constraint on <span class="math-container">$D$</span>), global normalization strategies based on dataset-level statistics <span class="math-container">$\mu_c$</span>, <span class="math-container">$\sigma_c$</span> rather than per-image min-max scaling, and more advanced domain adaptation that aligns color distributions in feature space.</p>
  <p>Clipping gradients, for instance
  <span class="math-container">$$
  \nabla_w L_{\mathrm{gen}} \leftarrow \frac{\nabla_w L_{\mathrm{gen}}}{\max \Bigl(1,\;\frac{\|\nabla_w L_{\mathrm{gen}}\|_2}{g_{\mathrm{clip}}}\Bigr)},
  $$</span>
  can also prevent runaway updates. Introducing new-camera images progressively helps the model adapt to the updated manifold without abrupt changes that trigger unstable discriminator feedback. Checking for large intermediate activations by printing or asserting <span class="math-container">$\mathrm{isnan}(\mathrm{activation})$</span> or <span class="math-container">$\mathrm{isinf}(\mathrm{activation})$</span> at various layers can reveal exactly where numerical instability arises.</p>
  <p>Viewing this from a theoretical standpoint, abruptly shifting from one color distribution to another can drastically alter the empirical data manifold and cause blow-up in gradients due to unanticipated regions in parameter space, so the remedy involves careful alignment of distributions and robust training heuristics to ensure stable convergence in adversarial inpainting.​​​​​​​​​​​​​​​​</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-26T08:55:40.287Z
Answers.Text: I clamped the gradient between -1 and 1 before. But I still get a sudden nan in the iteration 4400```
                  sum(losses.values()).backward()
                  losses["advd"] = dis_loss
                  dis_loss.backward()
                  # Gradient clipping
                  torch.nn.utils.clip_grad_norm_(self.netG.parameters(), max_norm=1.0)  # Adjust max_norm as needed
                  torch.nn.utils.clip_grad_norm_(self.netD.parameters(), max_norm=1.0)  
                  self.optimG.step()
                  self.optimD.step()
  ``` images: ![Valid XHTML](https://imgur.com/a/pLQBH6R)
Answers.UserDisplayName: galoischan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T08:30:17.463Z
Answers.LastActivityDate: 2024-12-22T08:30:17.463Z
Answers.UserDisplayName: Alex Rudimer
Body: <p>I am now trying to train a GAN called AOT-GAN to do some inpainting operation on some anodized aluminium surfaces. At the beginning, I used a canon camera to take the photos for training the AOT-GAN.Although the white balance of those photos was a little bit wrong and the photos had some blueish color, the training of AOT-GAN went well and I got some good synthetic images.
  Example of image took by canon camera:</p>
  <p><a href="https://i.sstatic.net/zOdN4dX5.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zOdN4dX5.jpg" alt="canon image" /></a></p>
  <p>I later changed the camera. For this camera, the images had some greenish color and this time I used Bradford method to do a chromatic adaptation to the images and got some more realistic images.
  Example of image(chromatic adapted) took by a new camera:
  <a href="https://i.sstatic.net/829RHReT.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/829RHReT.jpg" alt="chromatic adapted image0" /></a></p>
  <p>However, once I used the new, realistic-looking images for training the AOT-GAN, the loss function of the AOT-GAN will become NaN after some iternation.
  I have already investigated following possibility:</p>
  <ol>
  <li>There are NaN in the data.</li>
  </ol>
  <p>Not likely. I have already written the following code for finding any possible NaN in the image, but I never saw that is activated. Furthermore, when I use a smaller learning rate, the sudden NaN will appear much later(~6600 iterations or even 10000 iterations). So I think it is likely that the introduction of data from new camera changed the manifold.</p>
  <p>Code for finding NaN input:</p>
  <pre><code>images_masked = (images * (1 - masks).float()) + masks
          # Check for NaN values in images and masks
          if torch.isnan(images).any():
              print(f&quot;NaN values found in images at iteration {self.iteration}&quot;)
          if torch.isnan(masks).any():
              print(f&quot;NaN values found in masks at iteration {self.iteration}&quot;)
  </code></pre>
  <p>L1 and adversial loss when learning rate is 1e-6:</p>
  <p><a href="https://i.sstatic.net/HRS6xXOy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HRS6xXOy.png" alt="L1" /></a></p>
  <p>Result(I used data argumentation, so the image has different color):</p>
  <p><a href="https://i.sstatic.net/cwdEI4Bg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cwdEI4Bg.png" alt="result when learning rate is 1e-6" /></a>
  2.    Those input data is not normalization.</p>
  <p>I normalized the input within the range of -1 and 1 by writing my own transform, but the the loss function still go nan.</p>
  <p>Code for doing Normalization:</p>
  <pre><code>class ToTensorCustom(A.BasicTransform):
  &quot;&quot;&quot;Convert image and mask to `torch.Tensor`
  * Image numpy: [H, W, C] -&gt; Image tensor: [C, H, W]
  * Mask numpy: [H, W, 1] -&gt; Mask tensor: [1, H, W]
  &quot;&quot;&quot;
  def __init__(self, always_apply=True, p=1.0):
      super().__init__(always_apply=always_apply, p=p)
  
  @property
  def targets(self):
      return {&quot;image&quot;: self.apply, &quot;mask&quot;: self.apply_to_mask}
  
  def apply(self, img, **params):
      &quot;&quot;&quot;Image from numpy [H, W, C] to tensor [C, H, W]&quot;&quot;&quot;
      default_float_dtype = torch.get_default_dtype()
      img_tensor = torch.from_numpy(img.transpose(2, 0, 1))
      if isinstance(img_tensor, torch.ByteTensor):
          return img_tensor.to(dtype=default_float_dtype)
      else:
          return img_tensor
  
  def apply_to_mask(self, mask, **params):
      &quot;&quot;&quot;Mask from numpy [H, W] to tensor [1, H, W]&quot;&quot;&quot;
      default_float_dtype = torch.get_default_dtype()
      # Adding channel to first dim if mask has no channel
      if mask.ndim == 2:
          mask = np.expand_dims(mask, axis=0)
      # Transposing channel to channel first if mask has channel
      elif mask.ndim == 3:
          # [H, W, C] to tensor [C, H, W] in case mask has C &gt; 1
          mask = mask.transpose(2, 0, 1)
      else:
          raise ValueError('Mask should have shape [H, W] without, '
                           'channel however provided mask shape was: '
                           '{}'.format(mask.shape))
      # To numpy
      mask_tensor = torch.from_numpy(mask)
      if isinstance(mask_tensor, torch.ByteTensor):
          return mask_tensor.to(dtype=default_float_dtype).div(255)
      else:
          return mask_tensor
  class NormalizeByMinMax(A.BasicTransform):
      &quot;&quot;&quot;Normalize image by its min and max values.&quot;&quot;&quot;
      def __init__(self, always_apply=True, p=1.0):
          super().__init__(always_apply=always_apply, p=p)
  
      def apply(self, img, **params):
          min_val = img.min()
          max_val = img.max()
          # Avoid division by zero
          if max_val - min_val &gt; 0:
              img = (img - min_val) / (max_val - min_val)
          return img.astype(np.float32)
  
      @property
      def targets(self):
          return {&quot;image&quot;: self.apply}
  
  class InpaintingData(Dataset):
  
      def __init__(self, args):
          super(Dataset, self).__init__()
          self.w = self.h = args.image_size
          self.mask_type = args.mask_type
  
          # image and mask
          self.image_path = []
          for ext in [&quot;*.jpg&quot;, &quot;*.png&quot;]:
              self.image_path.extend(glob(os.path.join(args.dir_image, args.data_train, ext)))
          self.mask_path = glob(os.path.join(args.dir_mask, args.mask_type, &quot;*.png&quot;))
  
  
  
  
          #mean_rgb, std_rgb = calculate_mean_std_per_channel(full_loader, len(dataset))
  
          # augmentation
          self.train_transform = A.Compose(
              [
                  A.HorizontalFlip(p=0.5),
                  A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=45, p=0.5),
                  A.RGBShift(r_shift_limit=25, g_shift_limit=25, b_shift_limit=25, p=0.5),
                  A.ColorJitter(0.05,0.05,0.05,0.05),
                  A.RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3, p=0.5),
                  NormalizeByMinMax(),
                  ToTensorCustom(),
              ]
          )
  </code></pre>
  <p>Loss function:<a href="https://i.sstatic.net/oJG0fHaA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oJG0fHaA.png" alt="enter image description here" /></a></p>
  <p>direct Output from the AOT-GAN,inpainting-ed images, original image(with data argumentation):<a href="https://i.sstatic.net/65t1bAjB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/65t1bAjB.png" alt="Output from the AOT-GAN" /></a><a href="https://i.sstatic.net/zOUQ0He5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zOUQ0He5.png" alt="inpainting" /></a><a href="https://i.sstatic.net/JPFiaH2C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JPFiaH2C.png" alt="original images" /></a></p>
  
Tags: <generative-adversarial-networks><objective-functions>
Title: sudden NaN in the loss function of training a GAN for inpainting(AOT-GAN) I am sure there is no Nan in the input
UserDisplayName: galoischan
ThreadText: # sudden NaN in the loss function of training a GAN for inpainting(AOT-GAN) I am sure there is no Nan in the input
  
  (Asked by: username_0 on 2024-12-21)
  
  I am now trying to train a GAN called AOT-GAN to do some inpainting operation on some anodized aluminium surfaces. At the beginning, I used a canon camera to take the photos for training the AOT-GAN.Although the white balance of those photos was a little bit wrong and the photos had some blueish color, the training of AOT-GAN went well and I got some good synthetic images.
  Example of image took by canon camera:
  
  [![canon image](https://i.sstatic.net/zOdN4dX5.jpg)](https://i.sstatic.net/zOdN4dX5.jpg)
  
  I later changed the camera. For this camera, the images had some greenish color and this time I used Bradford method to do a chromatic adaptation to the images and got some more realistic images.
  Example of image(chromatic adapted) took by a new camera:
  [![chromatic adapted image0](https://i.sstatic.net/829RHReT.jpg)](https://i.sstatic.net/829RHReT.jpg)
  
  However, once I used the new, realistic-looking images for training the AOT-GAN, the loss function of the AOT-GAN will become NaN after some iternation.
  I have already investigated following possibility:
  
  1. There are NaN in the data.
  
  Not likely. I have already written the following code for finding any possible NaN in the image, but I never saw that is activated. Furthermore, when I use a smaller learning rate, the sudden NaN will appear much later(~6600 iterations or even 10000 iterations). So I think it is likely that the introduction of data from new camera changed the manifold.
  
  Code for finding NaN input:
  
  ```
  images_masked = (images * (1 - masks).float()) + masks
          # Check for NaN values in images and masks
          if torch.isnan(images).any():
              print(f"NaN values found in images at iteration {self.iteration}")
          if torch.isnan(masks).any():
              print(f"NaN values found in masks at iteration {self.iteration}")
  
  ```
  
  L1 and adversial loss when learning rate is 1e-6:
  
  [![L1](https://i.sstatic.net/HRS6xXOy.png)](https://i.sstatic.net/HRS6xXOy.png)
  
  Result(I used data argumentation, so the image has different color):
  
  [![result when learning rate is 1e-6](https://i.sstatic.net/cwdEI4Bg.png)](https://i.sstatic.net/cwdEI4Bg.png)
  2. Those input data is not normalization.
  
  I normalized the input within the range of -1 and 1 by writing my own transform, but the the loss function still go nan.
  
  Code for doing Normalization:
  
  ```
  class ToTensorCustom(A.BasicTransform):
  """Convert image and mask to `torch.Tensor`
  * Image numpy: [H, W, C] -> Image tensor: [C, H, W]
  * Mask numpy: [H, W, 1] -> Mask tensor: [1, H, W]
  """
  def __init__(self, always_apply=True, p=1.0):
      super().__init__(always_apply=always_apply, p=p)
  
  @property
  def targets(self):
      return {"image": self.apply, "mask": self.apply_to_mask}
  
  def apply(self, img, **params):
      """Image from numpy [H, W, C] to tensor [C, H, W]"""
      default_float_dtype = torch.get_default_dtype()
      img_tensor = torch.from_numpy(img.transpose(2, 0, 1))
      if isinstance(img_tensor, torch.ByteTensor):
          return img_tensor.to(dtype=default_float_dtype)
      else:
          return img_tensor
  
  def apply_to_mask(self, mask, **params):
      """Mask from numpy [H, W] to tensor [1, H, W]"""
      default_float_dtype = torch.get_default_dtype()
      # Adding channel to first dim if mask has no channel
      if mask.ndim == 2:
          mask = np.expand_dims(mask, axis=0)
      # Transposing channel to channel first if mask has channel
      elif mask.ndim == 3:
          # [H, W, C] to tensor [C, H, W] in case mask has C > 1
          mask = mask.transpose(2, 0, 1)
      else:
          raise ValueError('Mask should have shape [H, W] without, '
                           'channel however provided mask shape was: '
                           '{}'.format(mask.shape))
      # To numpy
      mask_tensor = torch.from_numpy(mask)
      if isinstance(mask_tensor, torch.ByteTensor):
          return mask_tensor.to(dtype=default_float_dtype).div(255)
      else:
          return mask_tensor
  class NormalizeByMinMax(A.BasicTransform):
      """Normalize image by its min and max values."""
      def __init__(self, always_apply=True, p=1.0):
          super().__init__(always_apply=always_apply, p=p)
  
      def apply(self, img, **params):
          min_val = img.min()
          max_val = img.max()
          # Avoid division by zero
          if max_val - min_val > 0:
              img = (img - min_val) / (max_val - min_val)
          return img.astype(np.float32)
  
      @property
      def targets(self):
          return {"image": self.apply}
  
  class InpaintingData(Dataset):
  
      def __init__(self, args):
          super(Dataset, self).__init__()
          self.w = self.h = args.image_size
          self.mask_type = args.mask_type
  
          # image and mask
          self.image_path = []
          for ext in ["*.jpg", "*.png"]:
              self.image_path.extend(glob(os.path.join(args.dir_image, args.data_train, ext)))
          self.mask_path = glob(os.path.join(args.dir_mask, args.mask_type, "*.png"))
  
          #mean_rgb, std_rgb = calculate_mean_std_per_channel(full_loader, len(dataset))
  
          # augmentation
          self.train_transform = A.Compose(
              [
                  A.HorizontalFlip(p=0.5),
                  A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=45, p=0.5),
                  A.RGBShift(r_shift_limit=25, g_shift_limit=25, b_shift_limit=25, p=0.5),
                  A.ColorJitter(0.05,0.05,0.05,0.05),
                  A.RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3, p=0.5),
                  NormalizeByMinMax(),
                  ToTensorCustom(),
              ]
          )
  
  ```
  
  Loss function:[![enter image description here](https://i.sstatic.net/oJG0fHaA.png)](https://i.sstatic.net/oJG0fHaA.png)
  
  direct Output from the AOT-GAN,inpainting-ed images, original image(with data argumentation):[![Output from the AOT-GAN](https://i.sstatic.net/65t1bAjB.png)](https://i.sstatic.net/65t1bAjB.png)[![inpainting](https://i.sstatic.net/zOUQ0He5.png)](https://i.sstatic.net/zOUQ0He5.png)[![original images](https://i.sstatic.net/JPFiaH2C.png)](https://i.sstatic.net/JPFiaH2C.png)
  
  **Tags:** generative-adversarial-networks, objective-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-22. Score: 1
  
  The sudden appearance of NaN values in the loss function after introducing chromatically adapted images suggests that the distribution of training data has changed in a way that causes numerical instabilities during gradient updates. Consider a generator $G$ (for inpainting) and a discriminator $D$ (for distinguishing real from generated samples). A typical generator objective is
  
  $$L\_{\mathrm{gen}}(G) = \alpha\,L\_{\mathrm{rec}}(G) + \beta\,L\_{\mathrm{adv}}(G),$$
  
  where $L\_{\mathrm{rec}}(G)$ might be an $L\_1$ inpainting loss of the form
  $$
  L\_{\mathrm{rec}}(G) = \frac{1}{N}\sum\_{i=1}^N \bigl\lVert (G(x\_i,m\_i) - x\_i)\odot (1-m\_i)\bigr\rVert\_1,
  $$
  and $L\_{\mathrm{adv}}(G)$ can be
  $$
  L\_{\mathrm{adv}}(G) = -\frac{1}{N}\sum\_{i=1}^N \log\bigl(D(x\_i',\,G(x\_i,m\_i))\bigr),
  $$
  where $x\_i$ is the real image, $m\_i$ is the mask, and $x\_i'$ might be a ground truth version of $x\_i$. The discriminator objective $L\_{\mathrm{disc}}$ often takes the form
  $$
  L\_{\mathrm{disc}}(D) = -\frac{1}{N}\sum\_{i=1}^N \Bigl[\log\bigl(D(x\_i',\,x\_i)\bigr) + \log\bigl(1 - D(x\_i',\,G(x\_i,m\_i)))\Bigr].
  $$
  
  If the new images shift color statistics sufficiently, the discriminator may produce unbounded gradients that cause floating-point overflows in weight updates. A smaller learning rate delays but does not prevent this if $\|\nabla\_w L\_{\mathrm{gen}}\|$ can still grow large. Techniques that mitigate the domain shift and dampen exploding gradients can stabilize training, such as gradient penalties (like a Lipschitz constraint on $D$), global normalization strategies based on dataset-level statistics $\mu\_c$, $\sigma\_c$ rather than per-image min-max scaling, and more advanced domain adaptation that aligns color distributions in feature space.
  
  Clipping gradients, for instance
  $$
  \nabla\_w L\_{\mathrm{gen}} \leftarrow \frac{\nabla\_w L\_{\mathrm{gen}}}{\max \Bigl(1,\;\frac{\|\nabla\_w L\_{\mathrm{gen}}\|\_2}{g\_{\mathrm{clip}}}\Bigr)},
  $$
  can also prevent runaway updates. Introducing new-camera images progressively helps the model adapt to the updated manifold without abrupt changes that trigger unstable discriminator feedback. Checking for large intermediate activations by printing or asserting $\mathrm{isnan}(\mathrm{activation})$ or $\mathrm{isinf}(\mathrm{activation})$ at various layers can reveal exactly where numerical instability arises.
  
  Viewing this from a theoretical standpoint, abruptly shifting from one color distribution to another can drastically alter the empirical data manifold and cause blow-up in gradients due to unanticipated regions in parameter space, so the remedy involves careful alignment of distributions and robust training heuristics to ensure stable convergence in adversarial inpainting.​​​​​​​​​​​​​​​​
  
  > **Comments:**
  > **username_0**: I clamped the gradient between -1 and 1 before. But I still get a sudden nan in the iteration 4400```                 sum(losses.values()).backward()                 losses["advd"] = dis_loss                 dis_loss.backward()                 # Gradient clipping                 torch.nn.utils.clip_grad_norm_(self.netG.parameters(), max_norm=1.0)  # Adjust max_norm as needed                 torch.nn.utils.clip_grad_norm_(self.netD.parameters(), max_norm=1.0)                   self.optimG.step()                 self.optimD.step() ``` images: ![Valid XHTML](https://imgur.com/a/pLQBH6R)  
  
  ---
  
  