I made sure its contiguous too
Here is what I did.
device = ...
ranks = [....]
saved = [None for rank in ranks]
a = saved[rank]
if a is not None and b.size() == a.size() and b.storage_offset() == a.storage_offset() and b.stride() == a.stride() and b.is_contiguous() and a.is_contiguous():
    # no need to call as_strided_
    a.copy_(b)
else:
    a = b.to(device)
    saved[rank] = a

when we replace that ugly if with if False:  everything works.
(what happens next is sending a through queue and cloning the the receiver, and then normal neural net)