Different optimisers require different amounts of memory. Normal SGD only involves the gradients, but if you want to store the momentum of the weights this is going to double the memory requirement because of the optimisation step. If you look at the Adam code, you’ll see it literally allocates the two buffers I mentioned at the first call of optim.step(), so different bits can be allocated at different parts of the overall optimisation. I’ll have to leave practices of model parallelism to someone with more than one GPU though~