Could you try to narrow down the issue a bit?
E.g. remove the custom convolution for multi GPU training and see, if you still get the OOM error.
Swapping custom classes for vanilla PyTorch ones might give us more information, which module leaks the memory.
Also, are you using nn.DataParallel, DDP or a custom implementation?