Hi,
My first question would be why can’t you run the second network in a batched manner? You could create all the combinations of batch by doing something like: batchwise_descriptors.unsqueeze(0).expand(full_size) for the first one and batchwise_descriptors.unsqueeze(1).expand(full_size) for the second one. Where full_size is (batch_size, batch_size, your_other_dims...).
That would definitely be the most efficient in terms of speed.
Not if you have hard constraints that you cannot do that (or memory issues), you can do the following:
(I haven’t tested the code so there might be a typo, but this is the spirit).
batch = get_new_batch()
optim.zero_grad() # Make sure all .grads are 0
batchwise_descriptors = learnable_network_1(batch)

# Create a new leaf Tensor that will accumulate all the grads
# coming from net 2
net_2_in = batchwise_descriptors.detach().requires_grad_()

total_loss = 0
for i in range(batch.shape[0]):
    for j in range(i+1, batch.shape[0]):
        pseudo_distances = learnable_network_2(
              net_2_in[i],
              net_2_in[j]
         )
        local_loss = loss_fn(pseudo_distances)
        local_loss.backward() # Accumulate in in learnable_network_2 and net_2_in gradients
        total_loss += loss.detach() # Just used for bookeeping, no gradient should flow back

# Now backwarprop the whole thing in net 1
batchwise_descriptors.backward(net_2_in.grad)
optim.step()