I have removed
       nn.ReLU(),
       nn.Sigmoid()

on the encoder end and now the loss is in rather normal range and decreasing.
If anyone has the same problem or is creating an unsupervised autoencoder