Does the memory error occur directly when starting the training or after some little time? If the latter, maybe there’s a memory leak in the training itself.
Did you try with (much) smaller batch sizes?