Speedup using NVIDIA Apex 
Hi,

According to PR https://github.com/huggingface/pytorch-pretrained-BERT/pull/116, we should be able to achieve a 3-4 x speed up for both bert-base and bert-large. However, I can only achieve 2x speed up with  bert-base. My docker image uses CUDA9.0 while the discussion in the PR https://github.com/huggingface/pytorch-pretrained-BERT/pull/116 is based on CUDA10.0... I am wondering if that makes the difference....

Thanks
Maybe. You can try with pytorch docker image `dockerhub 1.0-cuda10.0-cudnn7` to debug, as we did in the discussion in PR #116.<|||||>Just verified that CUDA10.0 makes 4x speedup. It should be better to include this in the main document.<|||||>What GPU do you run ... and how do you increase such a speedup? Is this possible with gtx 1080?

Thanks