I think the error is raised, since model.classifier[0] might not have been pushed to the device.
While you are using model = models.vgg().to(device) correctly, you are manipulating the classifier later without calling .to() again on this part of the model, so it should stay on the CPU.