Thanks for the pointer. I checked and as far as I can tell I’m not keeping a variable alive for more than it should be.  Explicitly deleting variables does seem to help, but shouldn’t garbage collection immediately free dereferenced memory anyway?
A follow-up is whether nvidia-smi accurately reports memory usage? I managed to run my training session on a Tesla K20 (4G memory). When I run the same training (same data and batching) on a 10G gpu nvidia-smi says I’m using 9G! Clearly, this can’t be because I just successfully ran it on a 4G GPU.