Apologies for being vague. I had trained an autoencoder model on cuda:1, and then ran the equivalent of the second code section above. The reconstructed image from the autoencoder was not correct, it looked as if the wrong batchnorm were being applied. When I changed the code to the first section it worked fine.
I’m sorry, I don’t have a tidy minimum example of the issue yet, but if you think it sounds suspicious I would be happy to try to make one.