Hi all,
I have a trained model in Python but ultimately want to run inference in C++.  I followed the procedure to save a serialized model with Torchscript.  I then load the model and then put it into the GPU.
I am working in a pipeline where all data resides in the GPU.  Now my question is if it is possible to take advantage of the fact that the data is already in the GPU and somehow convert this to a PyTorch GPU tensor, without having to perform and host/device copies.  I believe I have found an example of something similar: https://github.com/pytorch/pytorch/issues/19786 6
The link references a post that shows how to convert a cv::cuda::GpuMat of type 32F to a torch::Tensor.  This isn’t exactly what I would like to do as I am not working with cv::cuda::GpuMat types, but the data resides in Cuda memory and I believe the concept is similar.  Essentially the method torch::from_blob is used for the conversion.
I have also found another post: Constructing PyTorch's CUDA tensor from C++ with image data already on GPU 14
However, this post does not seem to reference torch::from_blob.  Rather, talks about using pointer references and using the cudaMemCpyAsync method.  However, unless I am mistaken I believe this does do a host/device copy.
Am I on the right track here?