Hello,
I was trying to train a model net that creates embeddings of images and a classifier clas  that discriminates the class of two embbedings. My training routine tries to learn both the embedder and the classifier on a end-to-end fashion.
At testing time, the model does not learn anything and I’m not sure why.
The code that I use is shown below:
net = Embedder()
net.train()
net.to(device)

clas = Classifier()
clas.train()
clas.to(device) 

optimizer = optim.Adam(list(net.parameters()) + list(clas.parameters()), lr=lr) 

criterion = nn.CrossEntropyLoss()

The training loop is:
images1 = images1.to(device, dtype=torch.float)
images2 = images2.to(device, dtype=torch.float)
labels = labels.to(device, dtype=torch.long)

# ================= FORWARD=================

e1 = net(images1)
e2 = net(images2)
rta = classifier(e1, e2)

loss = criterion(rta, labels)

# ================= BACKWARD =================

optimizer.zero_grad()
loss.backward()
optimizer.step()

I think the problem is that each embedding is computed on different forwards but I’m not sure.
I already done hyper-parameters searching and checked that the dataloader is working fine. Also, I tested wich optimizer to use, non of them do.
Thanks for any advice <3!