Are you getting this error only when using the fairseq repository or also if you are using plain PyTorch code?
In the former case, I would recommend to create an issue in the fairseq repository, in the latter case, could you post an executable code snippet, so that we could reproduce this issue?