I’ve already tried SGD - it was 3-4 times faster, but convergence was much slower.
And I’ve noticed, that my train time increased when I have added weight decay.
But what I am wondering - why the same model architecture on the same data with same parameters, but on TF2 trains much faster (3-4 times) than my on PyTorch.