The performance gains depend on the operations, shapes, and other potential bottlenecks.
E.g. convolutions in cudnn7.2 and earlier needed to have input channels, output channels and batch size as multiple of 8 to be able to use TensorCores. This restriction was listed in cudnn7.3 and later.
GEMMs however should still use matrices with shapes of multiples of 8 to use TensorCores in cublas and cudnn.
Are you preloading the data or are you using e.g. a DataLoader?
Note that you might encounter new bottlenecks in your code, if the model was the previous one and now got accelerated.