I am happy, but the code is huge, and I dont have any idea where this behaviour may come from.
Maybe I can describe main features.
It is a convolutional neural network with

1 conv-layer (7 filters 11x11, stride and paddings are all equal to 5, no maxpools nor batchnorms) and
2 fully-connected layers (with the second layer being an output layer). I apply batchnorm to the first FC-layer.

I use ReLU activation functions everywhere.
The loss function is defined by torch.nn.MSELoss(), i.e. mean-squared-error loss function.
I use Adam optimization method.
Please, let me know if there is anything else that needs to be mentioned.
I tested my code on the MNIST hand-written digits dataset, and it worked well (provided that loss is cross-entropy type loss and few other corrections to a classification problem).