I have a convolutional autoencoder, where the output layer is:
        self.t_conv5 = nn.ConvTranspose2d(128, 3, 3, stride=2, padding=1, output_padding=1)
        ...
        x = torch.sigmoid(self.t_conv5(x))

and the loss criterion is given by nn.BCELoss(). When trying to utilise autocast for fp16 conversion, the error that BCELoss is unsafe is given and the program suggests using BCEWithLogits instead to combine the final sigmoidal layer and BCE loss. This obviously stacks another sigmoidal layer on top of the desired output. If I remove the sigmoidal activation and re-apply it to the model output (i.e. output = model(images) becomes output = torch.sigmoid(model(images)), the calculated loss after 20 epochs of training remains the same but there’s a drastic difference in the output images:
image1256×582 134 KB
But as this is the same data and shuffling is disabled, the rearrangement shouldn’t have changed the output in this way. Is there some loss of relevant weights by having to rearrange the location of the sigmoid function in this way, and is there some way to counteract this? The autocasting, to my knowledge, should not reduce the final accuracy.