Sorry for the delay in getting back. With modules I mean environment modules:
http://modules.sourceforge.net/ 1
I could find out that the reason the from source installation with slower was indeed that it didn’t use cuDNN. I could download cuDNN and set the respective pathes (CUDNN_LIBRARY and CUDNN_INCLUDE_DIR) before executing setup.py. However when loading an environment module and exporting the CUDA_HOME pytorch was not finding a cudnn file after installation even with the cudnn pathes set. When using a local installation of cuda (11.0 in our case) we didn’t have this problem.
Anyways we now have the computational speed we want, just the from source installation with cuda modules doesn’t seem to work.
Thanks for the help!