Hello everyone,
Recently, we are focusing on training with int8, not inference on int8. Considering the numerical limitation of int8, at first we keep all parameters in fp32 and only quantize convolution layer (conduct int8 operation) as it is the most compute-intensive part of a model. During the past months, we have achieved some progress (such accuracy comparable to fp32 training and faster than fp32 training), and our paper is accepted by CVPR2020 (https://arxiv.org/pdf/1912.12607.pdf 36).
Now, we want to further explore quantization of other layers like ReLU, Pooling and BatchNorm, and keep dataflow in int8 in order to save memory.
(1) However, we could not use int8 tensor as input & output of a layer in PyTorch due to the autograd mechanism. Would you consider supporting int8 tensor in the future?
(2) Moreover, we want to pass quantization parameters like scale from layer to layer, but there are some problems at the shortcut connection. During backward, the autograd mechanism will add the gradient from main path and the gradient from the shortcut connection automatically. As we map quantization parameters to tensor, the quantization parameters are lost after the add operation. Could you provide some suggestions?
Many thanks.