In terms of eval speed on CUDA, then would it be still a good idea to drop observers? or just disabling them would be sufficient?

You can use model.apply(torch.quantization.disable_observer) and model.apply(torch.quantization.enable_observer) to toggle them (example: https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py#L128 6).