Dear guru,
I’m working with pytorch 1.3.0
Here is a typical plot of train/test losses behaviour as epoch increases.

I’m not an expert but I have read several topics on similar problems. Well, let me explain what I’m doing.
First, I have used implementation given by https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py 18 for resnet18 & resnet50,  and by https://github.com/akamaster/pytorch_resnet_cifar10 13 for resnet32, resnet56. For all these nets I got the same kind of test-loss hieratic behaviour.
Second, my inputs are images 5x64x64, so I have adapted the first Convolutional Layer, and the output of the last Full-connected consist of 180 neurons. I have used either 64, 128, 256 batch sizes for the training, and 128 for the test: the same behaviour persists. I have also both used 300k or 100k images in input training (100k for the test): same  behaviour persists too. The images are not of “standard” RGB photos: first, as you probably have already, ,  remarked there are 5 channels, second the pixel values can be negative (eg. spanning the range  (-0.01, 500))
Third, I am aware of the model.train() statement for the training phase, as well as the model.eval() statement (coupled with the  with torch.no_grad() for the testing phase.  It is clear that if I do not use model.eval() during the test phase, the test loss is gently decrasing as the traing loss. But, this is not allowed, isn’t it?
I have tried several things after reading post concerning Batch Norm behaviour wo any success

I have used SGD, Adam (& SWATS)
I have tryied lr = 0.1 to lr= 1e-5
I have modified the BN momentum (default = 0.1) : 0.5 and 0.01

Now, I have managed to get nice results (ie; good training & testing losses) with a classical CNN (ie. wo any Batch normalization, & short-cuts) but I would like to study  Resnet behaviour against adversarial attack. So, I would like to get Resnet fit my images 
Any idea ?
Thanks