The original Inception model was trained using the auxiliary outputs (which are also provided in the torchvision implementation) to “increase the gradient signal” as explained in the paper 1.
If you are not using it, I would guess the gradient might vanish at some point.