So I tried to compile PyTorch from scratch with CUDA support. I installed CUDA toolkit 9.2 locally, configured the environment variables and compile-installed PyTorch to a clean conda environment (as described in the PyTorch repo).
Unfortunately, this seemed to not work either. CMake successfully detects CUDA and cudnn during compilation. Weirdly enough, after installation running
import torch
print(torch.cuda.is_available())
print(torch.version.cuda)
print(torch.backends.cudnn.enabled)
print()

torch.zeros(5).cuda()

prints
False
9.2.148
True

THCudaCheck FAIL file=../aten/src/THC/THCGeneral.cpp line=51 error=2 : out of memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yardima/miniconda3/lib/python3.7/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (2) : out of memory at ../aten/src/THC/THCGeneral.cpp:51

What could be wrong here? The CUDA 9.2 samples still compile and run cleanly, but PyTorch does not work.
As a side note, I tried compiling with CUDA 9.0, but CMake complained GCC version >6 is incompatible with CUDA <9.2. The server I use has GCC 6.3.0 but not GCC 5. Is there a way to circumvent this without going through the hassle of compiling GCC-5, as I do not have root priveleges?