Hi @ptrblck ,
Thank you for your prompt reply. I did some further digging with thanks to your information.
I think I have a few follow up questions if you do not mind:

Is there a way in torch internals to lower the VRAM necessary for the CUDA context?
(link to 1) Is there a way to remove unnecessary internal caching if I am 100% sure that the only torch tensor that will be allocated is the input tensor? As expected, I always see a lot of memory reserved that is not in use.
If PyCUDA has already created context, must we still create a new context for PyTorch on the same running script?

I have PyCUDA + TensorRT code in the same script with my PyTorch pre/post-processing code, the context has already been created:
import pycuda.driver as cuda
ctx = cuda.Device(0).make_context()

It seems that creating the context as shown above only takes up 200 MiB while PyTorch initialisation takes up to 770MiB. Tested on my GTX1080, CUDA11.1.
If there is no way to remove the memory caching/reserved in PyTorch Tensor then it seems that PyTorch can only be used for quick development to train and test models. Seems like the workflow has to be:
PyTorch Development → Get model weights → Convert to TensorRT → Convert all pre and post-processing to CUDA implementation (super time consuming)