Hello everyone,
I’m building an app that makes calculations using CUDA (it makes some optimization based on  Simulated annealing). I successfully followed Custom C++ and CUDA Extensions 4 tutorial and made stable version on simple GPU, so now I would like to use multiple GPUs (some tasks has huge amount of data, that could not be allocated a single GPU + I’d like to speed up my calculations).
I have several tensors that I would like to split by dim=0 and make distributed calculations (all calculations based on map pattern, so all records by dim=0 are undependable). So best choice for me would be create my custom nn.Module class with forward method and use DistributedDataParallel module, but I have not any parameter that requires a gradient and module crushes. (Yeah it rises AssertionError: DistributedDataParallel is not needed when a module doesn’t have any parameter that requires a gradient.)
Could you please recommend something how to solve this problem or some other modules/ways to have distributed calculations.
Best regards, Demetry