It looks like this part of the model is wrong:
  (decoder): Sequential(
    (0): ConvTranspose2d(512, 320, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

Since the transposed convolution will output 320 channels, while the following batchnorm layer expects 512 input channels.