So you mean this does not work?
device = ...
ranks = [....]
saved = [None for rank in ranks]
a = saved[rank]
if a is not None:
    a.copy_(b)
else:
    a = b.to(device)
    saved[rank] = a

What is the issue you see with this?