Sorry for the late reply. I tried scatter and confirmed that it is the function I was looking for.
I was misled by the scatter_nd in TensorFlow which has a more scattered capacity.
However, torch.scatter is just simple and handy with the gradients tracked.
Thank you!