Hey thank you so much for your support!
I could get max 69.9% validation accuracy with

random horizontal flip
normalization
random affine for horizontal and vertical translation
mixup(alpha=1.0)
cutout(num_holes=1, size=16)

Random crop was decreasing val_acc. I guess I shouldn’t use it with cutout.
Weight initialization didn’t effect substantially. Here is what I did:

Remove softmax layer
Remove weight decay from optimizer
Decay only conv2d and linear layer weights manually

When we just add weight decay to the optimizer, it decays all differentiable parameters including biases and learnable parameters of batch normalization layers. In my case this, plus using softmax layer had catastrophic effects.
Keras model had kernel initializers on conv2d and linear layers, hence it didn’t have such a problem. It also has softmax layer.