I have the same sort of problem. I’m implementing a copy mechanism and need to get all the probabilities into a larger matrix.
 attention_scores = Variable(torch.zeros(target_seq_len, batch_size, max_dict_index))
 for i in range(batch_size):
     for j in range(target_seq_len):
         for k in range(input_seq_len):
             attention_scores[j, i, input[k, i].data[0]] += attentions[j, i, k]

But this naturally doesn’t work because Variables don’t support in-place assignment. Is there another way to construct a complex matrix, as would be done with scatter, and still retain the autograd history?