Have you tried to reduce your augmentation variance?
E.g. You could start to create a first train/val dataset reducing the augmentation hyperparamters rangers.
If It work well you could try to extend the range a bit and so on.