This problem is not closely related to the language property of Pytorch.  If it is not proper to ask this question here, I will delete it right away.
However, I couldn’t think of anywhere that will have so many experts who are familiar with NN structure and design.

Recently, I tried to implement VINet[1] a visual inertial odometry system that built with the neural network, and I open source it to GitHub HTLife/VINet 21
I already complete whole network structure, but the network can’t converge properly in training.

c21c66fe-70d4-45a2-ba59-49f6d6e36196.jpg827×480 46.8 KB

How could I fix this problem?
Possible problems & solutions


The dataset is too challenging:
I’m using the EuRoC MAV dataset, which is more challenging than the KITTI VO Dataset used by the DeepVO, Vinet(because the KITTI vehicle image does not shake up and down). NN cannot learn camera movement correctly.


Loss function:
L1 loss is been used and identical to the design in [1]. (I’m not very confident about whether I understand the loss design in [1] currently.) Related code


Other hyperparameter problems


References

[1] Clark, Ronald, et al. “VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem.” AAAI. 2017.