ptrblck:

Try to remove the last relu activation in your model, which should make the output more diverse.


Thats the first thing I did just after posting, and from the limited testing I done so far, I’d say that the network is learning faster.



 ptrblck:

Also, I’m not sure if it’s a good idea to re-sample the data in the training loop, as your model won’t be able to learn these random samples.


I am afraid I don’t understand what you mean by re-sample the data.  My current training loop(using real data, not the random data example given above) looks like this:
for index, triplet in enumerate(data):
    optimizer.zero_grad()

    anchor = model(torch.tensor([[triplet[0]]]).float()
    positive = model(torch.tensor([[triplet[0]]]).float()
    negative = model(torch.tensor([[triplet[0]]]).float()

    loss = criterion(anchor, positive, negative)
    loss.backward()
    optimizer.step()

    # print error and others every 100 iters.

Thanks.