Are you manually casting the model to half() or any parts of it?
Also, could you update to the latest PyTorch release (1.9.0), if you are using an older one? In case you are still hitting this issue in the latest release, could you post an executable code snippet to reproduce this issue?