It could be true that the implementation for Adam is not very memory efficient. Regarding Adadelta vs Adam, the former only has one vector of the past gradients whereas the second has two parameters, beta1 and beta2 that are multiplied with two vectors, so that’s maybe explaining why it’s roughly twice as much memory.
Btw if you really need/want to use Adam on a net where such a problem occurs, you can try the new checkpointing feature (https://pytorch.org/docs/master/checkpoint.html 4 ; an example can be found in the release docs at https://github.com/pytorch/pytorch/releases/tag/v0.4.0 1 if you scroll to the section “Neural Networks”). Haven’t used it myself yet, and it will make your code slower, but it would be a workaround for now instead of using another optimizer.