This wouldn’t allow you to either rebuild PyTorch with GPU support or any custom CUDA extension, would it?
If I understand your use case correctly, you are installing the binaries with the CUDA runtime libs.