Since you don’t actually call backward, your graph doesn’t get destroyed. I think it would be better, to always backward and (depending on the value of your loss) call an additional zero_grad (or don’t call the optimizer step):
total_loss = criterion(anchor_f, pos_f, neg_f)        
optimizer.zero_grad()
total_loss.backward()
if total_loss.item() > 1e-6:
    optimizer.step()
else:
    optimizer.zero_grad()