FP16 operations can be accelerated using Tensor cores (or Volta cores), which were introduced in Volta and Turing GPUs (e.g. GTX 2080).
Linear layers use GEMMs to compute the output. The matrices involved in this computation must have sizes of multiples of 8 to use Tensor cores:
a = torch.randn(8, 16)
b = torch.randn(16, 32)
torch.matmul(a, b)

This requirement exists for all cublas and cudnn versions, as far as I know.
There were also other requirements for convolution layers for cudnn 7.2 and older.
The current PyTorch binaries ship with cudnn 7.6, so you don’t have to worry about it.
That being said, the posted times look quite bad.
Could you post the code you’ve used to profile your model?
Also, which GPU are you using?