Thank you for your message Soumith! I just created a repository that can potentially also serve as a reference for other people who are trying to create a CUDA extension: https://github.com/sniklaus/pytorch-extension 9
This extension simply computes the Hadamard product using a custom CUDA kernel. Everything works well if it is just being executed on a single graphics card but once more than one GPU is being used, the configuration seems to affect the execution. Please see the test.py file for more details.