I’m getting the following error when I try to backpropagate an error:
RuntimeError: The expanded size of the tensor (36) must match the existing size (3) at non-singleton dimension 0
(The 3 above is because I have it running in parallel on 3 GPUs. On a single GPU that becomes a 1.)
I suspect this has something to do with reductions, but I don’t quite understand why…
The above error only occurs when I add a new component to my error. That component is defined in a function that takes as input a batch of 36 variables, mutates them a bit (preserving their variable state), computes their distance from some ground truth, and returns a scalar: the average of the distances in the batch. It’s just an additional loss term.
The error occurs when I try to backpropagate as
one = torch.ones(num_gpus).cuda().double()

# ...compute err...
err = network(input)
# err.backward(one) <-- no problems here!

err = err + my_function(other_input, gt)

err.backward(one) # <--problems!

Note that everything is scalar by the time the backward computation is done:
print err.shape
>> ()

How do the loss functions that accept batches of variables address this? There doesn’t seem to be anything particularly obvious in the source code (here 1 or here). What is being “expanded” here and why is it causing this error?
I suspect this is because I am applying the function to a tensor of BATCH_SIZE x SAMPLE_ERROR rather than using a reduction that applies the function to each sample independently within the batch (a la reduce=True in the built-in losses). How can I resolve this?