Here is pseudocode
for input (X, Y)
W = nn.Parameter()
index = [1,1,1,2,3]
output = X*W[index]
loss = function(output, Y)
loss.backward()

In above case, parameter W[1] has three copies whose gradients will be computed independently as you say.  I hope the gradient of W[1] just computed once in loss.backward(). W[1]'s copies don’t need to compute the gradients, which can share the same gradient with W[1] . Otherwise the backward gets slow.