Pytorch does not support efficient INT8 scoring, and if you do not have Volta you will not gain any speed gain on both train and score on fp16. If you want fast scoring in int8 consider using of TensorRT you will get up to 3x faster scoring on ResNet like nets on INT8 with “slightly” lower accuracy