In  case it will help someone:
I actually do notice some deadlocks when combining to() and copy_() like mentioned above.
Deadlock happens inside (i.e compiled cuda code) of to() at the second call, that is
to(), copy_(), copy_(),…,copy_(), to(), deadlock.
Anyway, I got frustrated and now I’m using only to(), as its quite minor optimization I wasted too much time on.
I think its related to RTX2080ti not supporting p2p (I looked at the cuda code that does the to()  few weeks ago, and I think it assumes something about p2p, but I did not bother to check it thoroughly.)
I can’t share my code (yet), but as soon as I will I’ll share the full example