Loss is oscillating around some value but that is usually very large. Some more context below:
So, I have a pre-trained RNN encoder decoder architecture which extracts a feature vector from a sequence of images. I trained those using self-supervised learning (more details in this paper: https://arxiv.org/pdf/1909.04656.pdf) and I am confident they are extracting the correct features because their loss function (after just 5 epochs) drops to 1.5 or something.
So, in order to predict the bounding box coordinates of an agent (I am basically trying to do what nvidia is working on here: https://blogs.nvidia.com/blog/2019/05/22/drive-labs-predicting-future-motion/ 1), I extract features from the sequence of entire images, then I extract features from the sequence of the agent’s image and then I concatenate these two feature vectors with the most recent bounding box coordinates of the agent. This long vector has a length of 1028.
I put this one long vector through a fully connected layer of hidden size 512 and output size of 4. The outputs are not the raw bounding box coordinates in the future but delta values that are added into the input to get the future bounding box coordinates.
In every attempt, I noticed that my output would start with very small values (0.00012 etc) and then, within just 5 epochs they would get to huge values around 9.23 or -23.41 etc. for every agent and then they would just linger there and not change much at all.
I have double checked everything, I normalize the two feature vectors and the most recent bounding box coordinates separately before concatenating them (normalized using torch.nn.functional.normalize() and I double check that they are normalized by summing all elements along a row for each vector). I increase and decrease the learning rate (Adam and I also try having some weight decay but no change).
The only optimizer here is the optimizer for the fully connected layer which I set to zero before the main loop where I add up the values of all the loss functions before doing backprop outside the loop (I only do fc_optimizer.zero_grad() and not fc_model.zero_grad(), is there any difference between the two?)
Honestly, right now I am just trying a completely different approach since if my model can’t even overfit 150 images then the model simply can’t capture the relationship between the current input and desired output.