Yeah, found an updated example in apex 
Seems like latest apex is for pytorch-nightly and it requires a little more additional arguments related to memory format.
For now this looks like the best option but still hope memory format bindings could be handled only once at env setting.