I have the similar problem when model use the training - aware quantification and the training loss is not nan. How to solve it?
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “model-code/resnet34/train_age_quan.py”, line 1130, in forward
x = self.ConvBNReLU1(x)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward
input = module(input)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py”, line 243, in forward
return self.activation_post_process(F.relu(ConvBn2d._forward(self, input)))
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py”, line 95, in _forward
conv = self._conv_forward(input, self.weight_fake_quant(scaled_weight))
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/fake_quantize.py”, line 81, in forward
self.scale, self.zero_point = self.calculate_qparams()
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/fake_quantize.py”, line 76, in calculate_qparams
return self.activation_post_process.calculate_qparams()
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/observer.py”, line 481, in calculate_qparams
return self._calculate_per_channel_qparams(self.min_vals, self.max_vals)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/observer.py”, line 150, in _calculate_per_channel_qparams
), “min {} should be less than max {}”.format(min_vals[i], max_vals[i])
AssertionError: min nan should be less than max nan