Well to answer your question I realize that I certainly do not master every subtilities of ML 
Concerning the model Inception-like I mean it is a network with 1 conv layer followed by 5 inception cells followed by 3 fully connected layers. Each Inception cells gather 1x1, 3x3 and 5x5 eventually kernel convolutions processed in paralell. This is in fact the mode used by some colleagues: https://arxiv.org/pdf/1806.06607.pdf.
So, I find some Adam settings that is ok cf. the train and test loss derease epoch after epoch and I a certain point they diverge ie, overfitting is arround, so I stop the learning phase. As I save intermediate state it is quite easy. Now, I do not know what is the auxiliairy loss you are refeering too.
An other infomation I got today: with SGD I get the same behaviour.