<p>The sudden appearance of NaN values in the loss function after introducing chromatically adapted images suggests that the distribution of training data has changed in a way that causes numerical instabilities during gradient updates. Consider a generator <span class="math-container">$G$</span> (for inpainting) and a discriminator <span class="math-container">$D$</span> (for distinguishing real from generated samples). A typical generator objective is</p>
<p><span class="math-container">$$L_{\mathrm{gen}}(G) = \alpha\,L_{\mathrm{rec}}(G) + \beta\,L_{\mathrm{adv}}(G),$$</span></p>
<p>where <span class="math-container">$L_{\mathrm{rec}}(G)$</span> might be an <span class="math-container">$L_1$</span> inpainting loss of the form
<span class="math-container">$$
L_{\mathrm{rec}}(G) = \frac{1}{N}\sum_{i=1}^N \bigl\lVert (G(x_i,m_i) - x_i)\odot (1-m_i)\bigr\rVert_1,
$$</span>
and <span class="math-container">$L_{\mathrm{adv}}(G)$</span> can be
<span class="math-container">$$
L_{\mathrm{adv}}(G) = -\frac{1}{N}\sum_{i=1}^N \log\bigl(D(x_i',\,G(x_i,m_i))\bigr),
$$</span>
where <span class="math-container">$x_i$</span> is the real image, <span class="math-container">$m_i$</span> is the mask, and <span class="math-container">$x_i'$</span> might be a ground truth version of <span class="math-container">$x_i$</span>. The discriminator objective <span class="math-container">$L_{\mathrm{disc}}$</span> often takes the form
<span class="math-container">$$
L_{\mathrm{disc}}(D) = -\frac{1}{N}\sum_{i=1}^N \Bigl[\log\bigl(D(x_i',\,x_i)\bigr) + \log\bigl(1 - D(x_i',\,G(x_i,m_i)))\Bigr].
$$</span></p>
<p>If the new images shift color statistics sufficiently, the discriminator may produce unbounded gradients that cause floating-point overflows in weight updates. A smaller learning rate delays but does not prevent this if <span class="math-container">$\|\nabla_w L_{\mathrm{gen}}\|$</span> can still grow large. Techniques that mitigate the domain shift and dampen exploding gradients can stabilize training, such as gradient penalties (like a Lipschitz constraint on <span class="math-container">$D$</span>), global normalization strategies based on dataset-level statistics <span class="math-container">$\mu_c$</span>, <span class="math-container">$\sigma_c$</span> rather than per-image min-max scaling, and more advanced domain adaptation that aligns color distributions in feature space.</p>
<p>Clipping gradients, for instance
<span class="math-container">$$
\nabla_w L_{\mathrm{gen}} \leftarrow \frac{\nabla_w L_{\mathrm{gen}}}{\max \Bigl(1,\;\frac{\|\nabla_w L_{\mathrm{gen}}\|_2}{g_{\mathrm{clip}}}\Bigr)},
$$</span>
can also prevent runaway updates. Introducing new-camera images progressively helps the model adapt to the updated manifold without abrupt changes that trigger unstable discriminator feedback. Checking for large intermediate activations by printing or asserting <span class="math-container">$\mathrm{isnan}(\mathrm{activation})$</span> or <span class="math-container">$\mathrm{isinf}(\mathrm{activation})$</span> at various layers can reveal exactly where numerical instability arises.</p>
<p>Viewing this from a theoretical standpoint, abruptly shifting from one color distribution to another can drastically alter the empirical data manifold and cause blow-up in gradients due to unanticipated regions in parameter space, so the remedy involves careful alignment of distributions and robust training heuristics to ensure stable convergence in adversarial inpainting.​​​​​​​​​​​​​​​​</p>

CC BY-SA 4.0
2024-12-26T08:55:40.287Z
I clamped the gradient between -1 and 1 before. But I still get a sudden nan in the iteration 4400```
                sum(losses.values()).backward()
                losses["advd"] = dis_loss
                dis_loss.backward()
                # Gradient clipping
                torch.nn.utils.clip_grad_norm_(self.netG.parameters(), max_norm=1.0)  # Adjust max_norm as needed
                torch.nn.utils.clip_grad_norm_(self.netD.parameters(), max_norm=1.0)  
                self.optimG.step()
                self.optimD.step()
``` images: ![Valid XHTML](https://imgur.com/a/pLQBH6R)
galoischan
CC BY-SA 4.0
2024-12-22T08:30:17.463Z
2024-12-22T08:30:17.463Z
Alex Rudimer
<p>I am now trying to train a GAN called AOT-GAN to do some inpainting operation on some anodized aluminium surfaces. At the beginning, I used a canon camera to take the photos for training the AOT-GAN.Although the white balance of those photos was a little bit wrong and the photos had some blueish color, the training of AOT-GAN went well and I got some good synthetic images.
Example of image took by canon camera:</p>
<p><a href="https://i.sstatic.net/zOdN4dX5.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zOdN4dX5.jpg" alt="canon image" /></a></p>
<p>I later changed the camera. For this camera, the images had some greenish color and this time I used Bradford method to do a chromatic adaptation to the images and got some more realistic images.
Example of image(chromatic adapted) took by a new camera:
<a href="https://i.sstatic.net/829RHReT.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/829RHReT.jpg" alt="chromatic adapted image0" /></a></p>
<p>However, once I used the new, realistic-looking images for training the AOT-GAN, the loss function of the AOT-GAN will become NaN after some iternation.
I have already investigated following possibility:</p>
<ol>
<li>There are NaN in the data.</li>
</ol>
<p>Not likely. I have already written the following code for finding any possible NaN in the image, but I never saw that is activated. Furthermore, when I use a smaller learning rate, the sudden NaN will appear much later(~6600 iterations or even 10000 iterations). So I think it is likely that the introduction of data from new camera changed the manifold.</p>
<p>Code for finding NaN input:</p>
<pre><code>images_masked = (images * (1 - masks).float()) + masks
        # Check for NaN values in images and masks
        if torch.isnan(images).any():
            print(f&quot;NaN values found in images at iteration {self.iteration}&quot;)
        if torch.isnan(masks).any():
            print(f&quot;NaN values found in masks at iteration {self.iteration}&quot;)
</code></pre>
<p>L1 and adversial loss when learning rate is 1e-6:</p>
<p><a href="https://i.sstatic.net/HRS6xXOy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HRS6xXOy.png" alt="L1" /></a></p>
<p>Result(I used data argumentation, so the image has different color):</p>
<p><a href="https://i.sstatic.net/cwdEI4Bg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cwdEI4Bg.png" alt="result when learning rate is 1e-6" /></a>
2.    Those input data is not normalization.</p>
<p>I normalized the input within the range of -1 and 1 by writing my own transform, but the the loss function still go nan.</p>
<p>Code for doing Normalization:</p>
<pre><code>class ToTensorCustom(A.BasicTransform):
&quot;&quot;&quot;Convert image and mask to `torch.Tensor`
* Image numpy: [H, W, C] -&gt; Image tensor: [C, H, W]
* Mask numpy: [H, W, 1] -&gt; Mask tensor: [1, H, W]
&quot;&quot;&quot;
def __init__(self, always_apply=True, p=1.0):
    super().__init__(always_apply=always_apply, p=p)

@property
def targets(self):
    return {&quot;image&quot;: self.apply, &quot;mask&quot;: self.apply_to_mask}

def apply(self, img, **params):
    &quot;&quot;&quot;Image from numpy [H, W, C] to tensor [C, H, W]&quot;&quot;&quot;
    default_float_dtype = torch.get_default_dtype()
    img_tensor = torch.from_numpy(img.transpose(2, 0, 1))
    if isinstance(img_tensor, torch.ByteTensor):
        return img_tensor.to(dtype=default_float_dtype)
    else:
        return img_tensor

def apply_to_mask(self, mask, **params):
    &quot;&quot;&quot;Mask from numpy [H, W] to tensor [1, H, W]&quot;&quot;&quot;
    default_float_dtype = torch.get_default_dtype()
    # Adding channel to first dim if mask has no channel
    if mask.ndim == 2:
        mask = np.expand_dims(mask, axis=0)
    # Transposing channel to channel first if mask has channel
    elif mask.ndim == 3:
        # [H, W, C] to tensor [C, H, W] in case mask has C &gt; 1
        mask = mask.transpose(2, 0, 1)
    else:
        raise ValueError('Mask should have shape [H, W] without, '
                         'channel however provided mask shape was: '
                         '{}'.format(mask.shape))
    # To numpy
    mask_tensor = torch.from_numpy(mask)
    if isinstance(mask_tensor, torch.ByteTensor):
        return mask_tensor.to(dtype=default_float_dtype).div(255)
    else:
        return mask_tensor
class NormalizeByMinMax(A.BasicTransform):
    &quot;&quot;&quot;Normalize image by its min and max values.&quot;&quot;&quot;
    def __init__(self, always_apply=True, p=1.0):
        super().__init__(always_apply=always_apply, p=p)

    def apply(self, img, **params):
        min_val = img.min()
        max_val = img.max()
        # Avoid division by zero
        if max_val - min_val &gt; 0:
            img = (img - min_val) / (max_val - min_val)
        return img.astype(np.float32)

    @property
    def targets(self):
        return {&quot;image&quot;: self.apply}

class InpaintingData(Dataset):

    def __init__(self, args):
        super(Dataset, self).__init__()
        self.w = self.h = args.image_size
        self.mask_type = args.mask_type

        # image and mask
        self.image_path = []
        for ext in [&quot;*.jpg&quot;, &quot;*.png&quot;]:
            self.image_path.extend(glob(os.path.join(args.dir_image, args.data_train, ext)))
        self.mask_path = glob(os.path.join(args.dir_mask, args.mask_type, &quot;*.png&quot;))




        #mean_rgb, std_rgb = calculate_mean_std_per_channel(full_loader, len(dataset))

        # augmentation
        self.train_transform = A.Compose(
            [
                A.HorizontalFlip(p=0.5),
                A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=45, p=0.5),
                A.RGBShift(r_shift_limit=25, g_shift_limit=25, b_shift_limit=25, p=0.5),
                A.ColorJitter(0.05,0.05,0.05,0.05),
                A.RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3, p=0.5),
                NormalizeByMinMax(),
                ToTensorCustom(),
            ]
        )
</code></pre>
<p>Loss function:<a href="https://i.sstatic.net/oJG0fHaA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oJG0fHaA.png" alt="enter image description here" /></a></p>
<p>direct Output from the AOT-GAN,inpainting-ed images, original image(with data argumentation):<a href="https://i.sstatic.net/65t1bAjB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/65t1bAjB.png" alt="Output from the AOT-GAN" /></a><a href="https://i.sstatic.net/zOUQ0He5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zOUQ0He5.png" alt="inpainting" /></a><a href="https://i.sstatic.net/JPFiaH2C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JPFiaH2C.png" alt="original images" /></a></p>

<generative-adversarial-networks><objective-functions>
sudden NaN in the loss function of training a GAN for inpainting(AOT-GAN) I am sure there is no Nan in the input
galoischan
# sudden NaN in the loss function of training a GAN for inpainting(AOT-GAN) I am sure there is no Nan in the input

(Asked by: username_0 on 2024-12-21)

I am now trying to train a GAN called AOT-GAN to do some inpainting operation on some anodized aluminium surfaces. At the beginning, I used a canon camera to take the photos for training the AOT-GAN.Although the white balance of those photos was a little bit wrong and the photos had some blueish color, the training of AOT-GAN went well and I got some good synthetic images.
Example of image took by canon camera:

I later changed the camera. For this camera, the images had some greenish color and this time I used Bradford method to do a chromatic adaptation to the images and got some more realistic images.
Example of image(chromatic adapted) took by a new camera:

However, once I used the new, realistic-looking images for training the AOT-GAN, the loss function of the AOT-GAN will become NaN after some iternation.
I have already investigated following possibility:

There are NaN in the data.

Not likely. I have already written the following code for finding any possible NaN in the image, but I never saw that is activated. Furthermore, when I use a smaller learning rate, the sudden NaN will appear much later(~6600 iterations or even 10000 iterations). So I think it is likely that the introduction of data from new camera changed the manifold.
Code for finding NaN input:
images_masked = (images * (1 - masks).float()) + masks
        # Check for NaN values in images and masks
        if torch.isnan(images).any():
            print(f"NaN values found in images at iteration {self.iteration}")
        if torch.isnan(masks).any():
            print(f"NaN values found in masks at iteration {self.iteration}")

L1 and adversial loss when learning rate is 1e-6:

Result(I used data argumentation, so the image has different color):

2.    Those input data is not normalization.
I normalized the input within the range of -1 and 1 by writing my own transform, but the the loss function still go nan.
Code for doing Normalization:
class ToTensorCustom(A.BasicTransform):
"""Convert image and mask to `torch.Tensor`
* Image numpy: [H, W, C] -> Image tensor: [C, H, W]
* Mask numpy: [H, W, 1] -> Mask tensor: [1, H, W]
"""
def __init__(self, always_apply=True, p=1.0):
    super().__init__(always_apply=always_apply, p=p)

@property
def targets(self):
    return {"image": self.apply, "mask": self.apply_to_mask}

def apply(self, img, **params):
    """Image from numpy [H, W, C] to tensor [C, H, W]"""
    default_float_dtype = torch.get_default_dtype()
    img_tensor = torch.from_numpy(img.transpose(2, 0, 1))
    if isinstance(img_tensor, torch.ByteTensor):
        return img_tensor.to(dtype=default_float_dtype)
    else:
        return img_tensor

def apply_to_mask(self, mask, **params):
    """Mask from numpy [H, W] to tensor [1, H, W]"""
    default_float_dtype = torch.get_default_dtype()
    # Adding channel to first dim if mask has no channel
    if mask.ndim == 2:
        mask = np.expand_dims(mask, axis=0)
    # Transposing channel to channel first if mask has channel
    elif mask.ndim == 3:
        # [H, W, C] to tensor [C, H, W] in case mask has C > 1
        mask = mask.transpose(2, 0, 1)
    else:
        raise ValueError('Mask should have shape [H, W] without, '
                         'channel however provided mask shape was: '
                         '{}'.format(mask.shape))
    # To numpy
    mask_tensor = torch.from_numpy(mask)
    if isinstance(mask_tensor, torch.ByteTensor):
        return mask_tensor.to(dtype=default_float_dtype).div(255)
    else:
        return mask_tensor
class NormalizeByMinMax(A.BasicTransform):
    """Normalize image by its min and max values."""
    def __init__(self, always_apply=True, p=1.0):
        super().__init__(always_apply=always_apply, p=p)

    def apply(self, img, **params):
        min_val = img.min()
        max_val = img.max()
        # Avoid division by zero
        if max_val - min_val > 0:
            img = (img - min_val) / (max_val - min_val)
        return img.astype(np.float32)

    @property
    def targets(self):
        return {"image": self.apply}

class InpaintingData(Dataset):

    def __init__(self, args):
        super(Dataset, self).__init__()
        self.w = self.h = args.image_size
        self.mask_type = args.mask_type

        # image and mask
        self.image_path = []
        for ext in ["*.jpg", "*.png"]:
            self.image_path.extend(glob(os.path.join(args.dir_image, args.data_train, ext)))
        self.mask_path = glob(os.path.join(args.dir_mask, args.mask_type, "*.png"))




        #mean_rgb, std_rgb = calculate_mean_std_per_channel(full_loader, len(dataset))

        # augmentation
        self.train_transform = A.Compose(
            [
                A.HorizontalFlip(p=0.5),
                A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=45, p=0.5),
                A.RGBShift(r_shift_limit=25, g_shift_limit=25, b_shift_limit=25, p=0.5),
                A.ColorJitter(0.05,0.05,0.05,0.05),
                A.RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3, p=0.5),
                NormalizeByMinMax(),
                ToTensorCustom(),
            ]
        )

Loss function:
direct Output from the AOT-GAN,inpainting-ed images, original image(with data argumentation):

**Tags:** generative-adversarial-networks, objective-functions


---

## 1 answers

### Answer by username_1 on 2024-12-22. Score: 1

The sudden appearance of NaN values in the loss function after introducing chromatically adapted images suggests that the distribution of training data has changed in a way that causes numerical instabilities during gradient updates. Consider a generator $G$ (for inpainting) and a discriminator $D$ (for distinguishing real from generated samples). A typical generator objective is

$$L\_{\mathrm{gen}}(G) = \alpha\,L\_{\mathrm{rec}}(G) + \beta\,L\_{\mathrm{adv}}(G),$$

where $L\_{\mathrm{rec}}(G)$ might be an $L\_1$ inpainting loss of the form
$$
L\_{\mathrm{rec}}(G) = \frac{1}{N}\sum\_{i=1}^N \bigl\lVert (G(x\_i,m\_i) - x\_i)\odot (1-m\_i)\bigr\rVert\_1,
$$
and $L\_{\mathrm{adv}}(G)$ can be
$$
L\_{\mathrm{adv}}(G) = -\frac{1}{N}\sum\_{i=1}^N \log\bigl(D(x\_i',\,G(x\_i,m\_i))\bigr),
$$
where $x\_i$ is the real image, $m\_i$ is the mask, and $x\_i'$ might be a ground truth version of $x\_i$. The discriminator objective $L\_{\mathrm{disc}}$ often takes the form
$$
L\_{\mathrm{disc}}(D) = -\frac{1}{N}\sum\_{i=1}^N \Bigl[\log\bigl(D(x\_i',\,x\_i)\bigr) + \log\bigl(1 - D(x\_i',\,G(x\_i,m\_i)))\Bigr].
$$

If the new images shift color statistics sufficiently, the discriminator may produce unbounded gradients that cause floating-point overflows in weight updates. A smaller learning rate delays but does not prevent this if $\|\nabla\_w L\_{\mathrm{gen}}\|$ can still grow large. Techniques that mitigate the domain shift and dampen exploding gradients can stabilize training, such as gradient penalties (like a Lipschitz constraint on $D$), global normalization strategies based on dataset-level statistics $\mu\_c$, $\sigma\_c$ rather than per-image min-max scaling, and more advanced domain adaptation that aligns color distributions in feature space.

Clipping gradients, for instance
$$
\nabla\_w L\_{\mathrm{gen}} \leftarrow \frac{\nabla\_w L\_{\mathrm{gen}}}{\max \Bigl(1,\;\frac{\|\nabla\_w L\_{\mathrm{gen}}\|\_2}{g\_{\mathrm{clip}}}\Bigr)},
$$
can also prevent runaway updates. Introducing new-camera images progressively helps the model adapt to the updated manifold without abrupt changes that trigger unstable discriminator feedback. Checking for large intermediate activations by printing or asserting $\mathrm{isnan}(\mathrm{activation})$ or $\mathrm{isinf}(\mathrm{activation})$ at various layers can reveal exactly where numerical instability arises.

Viewing this from a theoretical standpoint, abruptly shifting from one color distribution to another can drastically alter the empirical data manifold and cause blow-up in gradients due to unanticipated regions in parameter space, so the remedy involves careful alignment of distributions and robust training heuristics to ensure stable convergence in adversarial inpainting.​​​​​​​​​​​​​​​​

> **Comments:**
> **username_0**: I clamped the gradient between -1 and 1 before. But I still get a sudden nan in the iteration 4400```                 sum(losses.values()).backward()                 losses["advd"] = dis_loss                 dis_loss.backward()                 # Gradient clipping                 torch.nn.utils.clip_grad_norm_(self.netG.parameters(), max_norm=1.0)  # Adjust max_norm as needed                 torch.nn.utils.clip_grad_norm_(self.netD.parameters(), max_norm=1.0)                   self.optimG.step()                 self.optimD.step() ``` images: ![Valid XHTML](https://imgur.com/a/pLQBH6R)  

---

