Hi, @tshrjn @ptrblck, it has been a long time but did you find any proper way to handle this problem?
Let’s say i have an aux. tensor with shape (N, D) and i have 4 GPU device. I’m thinking of repeating my aux. tensor in a way that i’ll have a new (repeated) tensor with shape (4N, D) and when this new tensor is shared, each GPU process will end up with my original tensor (with shape (N,D) obviously).
I’m not sure if this is feasible (I’ll try it) and I’m wondering if there is an easy solution. Thanks.