exactly.
I tried 2 networks.

WideResnet: it works fine with copy_(). all b tensors are contiguous there.
GPT2 (from huggingface): does not work. I know that some b tensors are non-contiguous.

For the GPT2 the task is zero-shot on wikitext2.
With to(device) I restore the perplexity from the paper (~29)
With copy_() the perplexity explodes.(high crazy numbers, like 8400, 121931 and so on)
I tired to look at the tensors with the debugger, they look fine.