So I am training a set of videos over two models. And I am logging my loss after each epoch. Both of the models are performing poorly wrt. loss. During my first epoch, the loss first dips and then increase and in the next epoch, it takes a step up. What can be the possible reasons? I am using Adam Optimizer and its multi-label classification.
Screenshot from 2020-04-11 03-09-35913×522 21.1 KB
 
Screenshot from 2020-04-11 03-10-00905×518 20.3 KB