Thanks for the kind reply! You are right, it is not a leak per se, but rather a very high memory requirement for ADAM. This does make sense, although it was not immediately obvious that ADAM would fail for a 2GB graph! Adadelta, Adamax, and Adam required > 9GB and ran out of memory on a AWS P2 instance (K80 w/ 11GB RAM).
Using another optimizer, however, does work. SGD adds 2GB. RMSprop adds about about 4GB. Curiously, Adagrad needs only an additional ~4GB as well (although nvidia-smi showed more). I’m surprised that Adagrad can run but Adam/Adamax/Adadelta fail, as I thought the data requirements were similar in terms of per-parameter alpha. Might this suggest that the implementations for those three are memory inefficient?