I get this error too:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-4d12a5af1b3f> in <module>
----> 1 trainer.run_epoch()

~/ccai/github/dev_omni/omnigan/omnigan/trainer.py in run_epoch(self)
    655                     param.requires_grad = True
    656 
--> 657                 self.update_D(multi_domain_batch)
    658 
    659             # -------------------------------

~/ccai/github/dev_omni/omnigan/omnigan/trainer.py in update_D(self, multi_domain_batch, verbose)
   1133                 d_loss = self.get_D_loss(multi_domain_batch, verbose)
   1134             self.grad_scaler_d.scale(d_loss).backward()
-> 1135             self.grad_scaler_d.step(self.d_opt)
   1136             self.grad_scaler_d.update()
   1137         else:

~/.conda/envs/omnienv/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py in step(self, optimizer, *args, **kwargs)
    287 
    288         if optimizer_state["stage"] is OptState.READY:
--> 289             self.unscale_(optimizer)
    290 
    291         assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer."

~/.conda/envs/omnienv/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py in unscale_(self, optimizer)
    238         found_inf = torch.full((1,), 0.0, dtype=torch.float32, device=self._scale.device)
    239 
--> 240         optimizer_state["found_inf_per_device"] = self._unscale_grads_(optimizer, inv_scale, found_inf, False)
    241         optimizer_state["stage"] = OptState.UNSCALED
    242 

~/.conda/envs/omnienv/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py in _unscale_grads_(self, optimizer, inv_scale, found_inf, allow_fp16)
    185                 if param.grad is not None:
    186                     if (not allow_fp16) and param.grad.dtype == torch.float16:
--> 187                         raise ValueError("Attempting to unscale FP16 gradients.")
    188                     else:
    189                         torch._amp_non_finite_check_and_unscale_(param.grad,

ValueError: Attempting to unscale FP16 gradients.

The piece of code yielding this error is:
with autocast():
    d_loss = self.get_D_loss(multi_domain_batch, verbose)
self.grad_scaler_d.scale(d_loss).backward()
self.grad_scaler_d.step(self.d_opt)
self.grad_scaler_d.update()

I’m using pytorch 1.6 and not calling half() on anything. Maybe this context can help: I’m training a GAN model and the exact same procedure on the generator’s loss, optimizer and scaler works without error.
Generator and Discriminator’s optimizers are Adam optimizers from torch.optim and grad_scaler_d and grad_scaler_g are GradScaler() instances from from torch.cuda.amp. @ptrblck where do I start debugging beyond looking for  .half()  calls?