Soo if you are using a custom dataset, how big is it compared to the original one- The paper says they have used samples of four consecutive frames resulting in 1059 samples(assuming 1059*4 frames). In my case, I am using 387 images + data augmentation per epoch. With data augmentations, I once generate 1059 frames and it still got stuck. Like you said the network might need more data or need to be reduced in size

Ofc I’m neither sure about the range of values your loss should be reaching but 10e-3 seems already small. I would bet it reached those values within the first epoch. - For 1e-3 it reaches the values in the first epoch and then the loss oscillates around the next values for a long time and diminishes very slowly
One of my last resort ideas was to eventually increase the training data to see if it helps in any way. However, using augmentation does the trick