When using DataParallel, is there is a way to set maximum allocated memory for single GPU?
When running my custom loss, additional memory gets allocated during the forward pass which is NOT only a few hundred megabytes. DataParallel is leaving some additional space on my GPU memory, but it’s not enough for running the forward loss.
Other than that, how does DataParallel decide how to split data?
Best,
Christian