Hi, as it has been unanswered for 2 months, how is it going with this? I cannot tell, what’s the issue here, but one general suggestion that has helped me tremendously in vision research is very simple: write unit tests, even for code that you think is plain simple and works OK. Very, very often the problem is not with a training code, but with data preprocessing, validation, evaluation etc. Definitely write unit tests for a loss function to see if it really does what you think it does 
All image and 3D transformation and normalisations are very hard to comprehend fully numerically, and it is very easy to make a small trivial error that renders the results unusable.
Also, plotting and visualising data after each processing step is also essential for debugging.