Yes, the model converges in float32. I have tried checking for NaNs and Infs with this code:
        scaler.step(optimizer)
        for param in model.parameters():
            if torch.isnan(param.data).any() or torch.isinf(param.data).any():
            	print(param.data)

but it doesn’t show anything. The loss is around 0.38 after the first step and then goes to NaN beacause the tensors returned by  out, latent_loss = model(img)  are filled with only NaNs.
The anomaly detection gives me “RuntimeError: Function ‘MseLossBackward’ returned nan values in its 0th output.”
I have isolated the problem and it seems to be in the Decoder since the NaNs appear after line 213
dec_t = self.dec_t(quant_t) in vqvae.py 3
Thanks for helping