In that case what about:

Perform sum reduction on the self: self=self.sum(0).sum(2)

Reshape index and src to delete obsolete dimensions: index = torch.reshape(index.permute(3, 0, 1, 2), (-1, index.size(2))
and src= torch.reshape(src.permute(3, 0, 1, 2), (-1, src.size(2))

Call the scatter add function of the already reduced self: self = self.scatter_add(0, index, src)

Unsqueeze the compressed dimensions: self = self.unsqueeze(0).unsqueeze(3)


Would that work/reduce memory print for you?
Samuel