You can try the resnet18 with the op InPlaceABN at this commit 3 . I train it on cifar10 using code here and pytorch 1.1.0, its validation accuracy is always lower than that of resnet18 with standard BatchNorm2d and there is a significant accuray drop during training:

result2048×1126 369 KB

How to fix this problem without modify the op since it is more memory efficient?