Hi,
I am surprised that it works in the forward, given that you index on dim2.
dim0 and dim1 should be the same for scatter but your dim1 is 1 in one case and channels in the other.
Could you try expanding idxs and vals to (batch_size, channels, N) before giving them to the scatter_add_? Does that fix the problem?