my guess is that somewhere in your code you have model.to(‘cuda’) (likely during training) and you are not converting it back to cpu i.e. model.to(‘cpu’) before trying to do quantization.

Strange because I have done model.to(‘cpu’) before torch.quantization.convert(model)