After hours and hours I found out that the problem is the input normalization (of the dataset): I use the normalization the with mean and std calculated over my specific dataset.
This leads to non-decreasing loss and producing always the same output (prediction) no matter the input.
If I use the standard [0.5, 0.5, 0.5] everything goes fine.
Do you have any explanation? I am a little confused…