You could use DLManagedTensor and fromDLPack 10 as an API or dive into the implementation 8 and use from_blob. Note that it means that you need to ensure the GPU memory blob is available for as long as the Tensor lives / until the deleter is called.
Best regards
Thomas