Try using a profiler, like the one embedded in pycharm, or single step through your code and time each call.
Two things to take notice in general:
Have you run backward propagation through unwanted tensors?
Have you vectorized your computation?