I’m doing a classification task with a training set of 20000 images over 1000 labels. I’m using Alex Net and triplet loss. The problem is that the loss usually stucks at the margin of triplet loss. I tried to adjust the learning rate from 0.01 to 0.000001 and momentum from 0.9 to 0.0009. Once it worked, the loss tends to converge to zero. But most of the time it doesn’t work even if I use the same setting as the time is worked. Can anyone tell me what shall I do?
Here’s the settings of my model.
model_color = AlexNet(3)
criterion = TripletLoss(margin=1.0)
optimizer_color = optim.SGD(model_color.parameters(), lr=0.00001, momentum=0.009)

I use online triplet generating method with batch hard strategy.
Here’s my training code. all_anchors_ contains all images in the current batch and get_positive will return the hardest positive and similar as get_negative.
for index, anchor in enumerate(all_anchors_):
    positive = get_positive(anchor)
    negative = get_negative(anchor)

    optimizer.zero_grad()

    a_output = model(anchor)
    p_output = model(torch.stack(positive))
    n_output = model(torch.stack(negative))

    loss = criterion(a_output, p_output, n_output)
    loss.backward()

    optimizer.step()

    running_loss += loss.data
    output = loss.data

I examine the loss.data and find that usually all of them stuck at 1, which is the margin of the triplet loss.