Hello PyTorchers.
I used the word PyTorcher meaning the person uses Pytorch.
I am not sure this expression is appropriate, so comment to me about this, please XD

For my task (face detection), I am using two deep learning framework, Pytorch and MXNet.
Please look at the diagram below to help your understanding.
image956×299 11 KB
When the code was first implemented, GPU was used for the process above.
But the inference time was not that fast.
To compare inference time, I later have created other codes that use CPU only(installed No CUDA libraries).
I found that there was no difference between GPU processing and CPU processing in terms of inference time. (I measure FPS but there was no dramatic difference.)

Could you give me ideas to use GPU efficiently for fast inference?
Is this normal that the Volatile GPU-util is almost 0% when PyTorch model is on inference mode?
I capture the RTSP frame using VideoStream class from imutils package which use cv2 and thread package. Do you have any idea for fast realtime ( < 2 sec delay) visualization?

I will definitely respond to all of your comments.
Really thank you so much in advance.