I cannot not find any NaNs in the foward pass when disabling autocast for the decoder, but the traceback of the error says it is still the decoder that produces NaNs, more specifically a convolution layer in the decoder:
\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
return self._conv_forward(input, self.weight)
lib\site-packages\torch\nn\modules\conv.py", line 420, in _conv_forward
self.padding, self.dilation, self.groups)
(function _print_stack)
I have modified the Sequential block into a line of code for each layer and the traceback now points to the first layer of the decoder which is a simple convolution: nn.Conv2d(in_channel, channel, 3, padding=1)