@ptrblck I think it runs for a few iterations with multiple GPUs but then crashes in the end of the first epoch. In the end of the epoch it seems to allocate more memory and then crashes. I also tried this with one GPU and cuDNN enabled and the same happens at about the same position.
Is there anything done in the background that I am not aware of?