How can I optimize PyTorch implementation?

I am not familiar with this particular implementation.
You can look online for “pytorch in production” and you should get all the materials you need.

Also, is it true that a fully optimized code in PyTorch will give the same results to that with Tensorflow’s OD?

This is not guaranteed. They will most likely be very similar because all the heavy lifting is done by cudnn and BLAS libraries anyway. But there might be some differences for less traditional architectures.

Also I think the main purpose of these frameworks is that we should more focus on implementation rather than optimization

I think this is a difference between pytorch and Tensorflow yes. While you can find many many things internalized and optimized in Tensorflow, it’s not the case for pytorch. We only add the very widely used methods to the core. You can find a bit more in the specialized libraries like torchvision or torchtext.