The problem here is on this line model.to(device). The .to()  method returns a pointer to the model stored in the specified device. Since you are not assigning this new pointer to any variable you are storing a model on GPU but still using the model stored in CPU. You should change the line to be like this:
model = model.to(device)
Hope that helps.