Further debugging leads to the operations applying torch.repeat on the input tensor.
I found that if I don’t use torch.repeat then it would function properly.
I was running something like this
psv_src = input_exts.reshape([-1, 4, 4]) # input_exts is [batch, #views, 4, 4]
psv_tgt = input_exts[:, 0:1].repeat([1, views, 1, 1]).reshape([-1, 4, 4])

trnfs = torch.matmul(psv_src, torch.inverse(psv_tgt))

So it looks like the repeat operation breaks something?
This only happens when the function is called again. The first time is working as expected.