I am working on crop disease with 20k images. I have imbalance dataset, so downsampled it but when I applied the resnet model on this I am getting training accuracy initially with  86%  but later epochs it is decreasing and val acc to 7% is constant with different batch_size, weight_decay. I have augmented data with brightness, rescaled, etc.