Regarding GPU, have you subtracted the time it takes to transfer data from main memory to the GPU?

Yes, 13seconds include that

Plus, your operations don’t seem to be well suited for taking advantage of GPUs

Oh, if that’s the case I’ll have to stick with numpy for this bit then. Is there a better implementation of NMS in Pytorch, I know a cuda NMS is being worked on, but anything else that I can use in the meantime?