Hi,
This is surprising indeed.
Do you see this gradient being non-zero after some time during training?
Also if you ignore gradients from one branch, is it non-zero?
# To ignore gradient from one branch, just detach it:
base = net(inp_base)
pos = net(inp_pos)
neg = net(inp_neg)

# To ignore gradients from the neg branch
neg = neg.detach()

loss = crit(base, pos, neg)
loss.backward()