I’m not sure the settings make sense, especially if cudnn.enabled = False is set. That being said, I’m not even sure what fraction of operations in a transformer like DistilMBERT would be dispatched to cuDNN, which should really only be doing the heavy lifting for convolution-heavy models.