I also tried removing weight decay from SGD, and adding it to conv and dense layers manually as given in here in-pytorch-how-to-add-l1-regularizer-to-activations 6 but still the validation accuracy is 50% with all listed regularizations. So it can’t be the weight decay.
I will really appreciate why with keras we can get 20% val_acc difference, but not with pytorch