Suspicion is that L1Loss is not differentiable. But in the NumPy code, the loss criterion is set to the difference without the absolute, so the loss could be back propagated.
Reference: http://christopher5106.github.io/deep/learning/2016/09/16/about-loss-functions-multinomial-logistic-logarithm-cross-entropy-square-errors-euclidian-absolute-frobenius-hinge.html 4
But then again I get the same failure rate when I change the criterion to SmoothL1Loss, 19 out of 100 times success rate…