@ptrblck Thank you very much for the suggestion.
I managed to make it work and noticed something strange during this process as mentioned below:
TensorRT tensor is showing such error when slicing in NumPy fashion. For example my_tensor[idx] is showing error but torch.index_select(my_tensor, 0, idx) is working fine.