maierf:

I also looped over all named parameters of my model and they all seem to be on GPU 0


That seems to be wrong, since some of your modules should be on GPU1.
As I’m currently unsure, what’s causing this issue, could you post an executable code snippet, so that we could reproduce it?