Using Fp16 half precision makes Bert prediction slower.
When I use:

     model = BertForMaskedLM.from_pretrained('bert-large-cased') 
     model = model.half()
     model.eval()
     model.to('cuda')

by adding Fp16:

     model = model.half()

It runs around 50% slower. Why is that?
 I run it on ubuntu 18.04, cuda 9, pytorch 1.1 and python 3.6.8



gtx 1080. Is there any other way to make the predictions faster?<|||||>Hi, you need at least a Volta GPU to get benefits from fp16 unfortunately.<|||||>@thomwolf  Does P100 applicable?<|||||>I don't think so<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
