Thanks for the help. I can’t say for sure what was going on but it may have been due to how I had my module set up (i also rebuilt from the latest source and updated cudnn). Basically I had a VAE with submodules encoder and decoder, and the VAE did the reparameterizing stuff and took care of other training boilerplate. After I moved everything to one monolithic module it seems to not be leaking memory anymore.
Or, the memory is ballooning but at a smaller rate, so that I can actually get through an epoch.