ah! Interesting!
Yes that indeed makes a difference.  Using RMSprop instead of Adam reduces the total allocated vram.  11.8 GB at loss.backwards() and 15.6 GB at optimizer.step().  So the optimization algorithm makes a difference.
Still not sure why optim.step() shows such a large increase though  still!  good to know!  Any idea how to distribute such a large model across multiple GPUs easily?  My initial thought was to manually place parts of the model on different devices and transfer the activation from device to device in forward(), but I am curious if there is a more elegant way.
Thanks!