Yeah, I meant the shapes etc. as setup, but fortunately the loss went down. Puh 
If the loss is still approx. 5x higher, your model might still learn something useful.
Could you try to run the training again? If nothing happens, you should weight this loss down a bit or adapt your learning rate.