Hi @ptrblck, once again, thank you for the prompt reply.
From your answers, I guess the only way is to figure out what library is the PyTorch functions specifically using (ex: for concat, stacking, NMS) and remove the rest during build (ex: cuDNN is not needed as DNN is managed by TensorRT).
Appreciate the help.

Potential future discussion for AI developers that stumbles upon this thread.
It seems that if we want to deploy using PyTorch functions (NMS, Stacking, etc) that made our lives so much easier, we have to deal with the huge memory allocated for its context.
This problem must be faced by many that want to deploy multiple modules, each leveraging PyTorch for ease of GPU programming. The context memory stacks up very quickly to fill up the entire VRAM of the GPU especially since deployment based GPU usually only has 8MiB-10MiB memory.
My personal workflow is that I use PyTorch for preprocessing and postprocessing as manipulating PyTorch Tensor is very easy.  The pointer of PyTorch processed Tensor (pycudatorch.py · GitHub 1) can then be passed into TensorRT (optimised model), output from TensorRT will remain as a PyTorch Tensor allowing very easy postprocessing (PyTorch readily available functions) and you can also use CUDA kernels that you have written to leverage the GPU parallelism (PyCUDA) when you need more control over the implementations of individual threads that PyTorch does not provide. The obvious weakness of this workflow is the PyTorch context memory allocation, I currently cannot run more than 2 models in my GPU due to this.
I guess the only way for developers that need to deploy multiple models/modules and are constrained by the GPU memory (usually the case) is to not use PyTorch at all for deployment.  Write functions you need in CUDA yourself and use TensorRT as usual.