Hello. I don’t know where else to ask this. Basically, I successfully extract features from a sequence of images and then concatenate this feature vector with a vector of bounding box coordinates and I put this long vector through a two-layer fully-connected layer to obtain bounding box coordinates of the future.
I wanted to see whether I can overfit a small subset of my data. So, I use this method on a sequence of just 150 images. I am unable to overfit it, in fact the loss function keeps increasing. I have done a lot to try to deal with this. I normalize the input to my fc-layer by using torch.nn.functional.normalize(). My fc-layer is a two-layer network which takes a 1028 vector as input and has one hidden layer with size 516 and the output is a vector of 4 values which are added to the input to predicit the future location of the bounding boxes.
Please tell me what kind of problems could I be facing here?  When something like this happens, does it mainly mean that there is no sensible relationship between the input and output or could some hyperparamter tuning fix this? (I also increased and decreased the learning rate using Adam without any weight decay)