The whole point of allocating every time is that it would allocate on the correct GPU.  If it’s only allocated once, then that would be option 4).  From my tests, it seems option 4) is about as fast as 2), so maybe I can just do that (assuming it works in multi GPU context…)
A checkpoint is saved every epoch, so deleting it before saving is a variation of 3).  I guess that’s an option.
Well, ideally this buffer is instantiated with the rest of the net in *.py, so “keep telling everyone those values” isn’t much of an issue.