Difference between this repo and bert-as-service
Hi, 

I wondered if anybody knows the difference between the `BertModel` of this repo and [bert-as-service](https://github.com/hanxiao/bert-as-service).

1. I cannot get the same result between these two even if I use the same checkpoint. pytorch-pretrained-BERT yield a lower acc and slower convergence.

2. The memory usage of `BertModel` seems much higher than bert-as-service. With the same batch-size=32, max_seq_len=100, the bert-as-service will take about 8000MB but `BertModel` will cost more than 16000MB because I got an OOM issue.

Does any one knows the reason behind it? Is there any optimization done for bert-as-service? 
Yes there are some optimizations. Please take a look here:

https://hanxiao.github.io/2019/01/02/Serving-Google-BERT-in-Production-using-Tensorflow-and-ZeroMQ/#engineering-building-a-scalable-service<|||||>Hi, there is no specific relation between the present repo (which provides PyTorch implementations of several transformer's models) and bert-as-a-service (which is built on top of BERT TensorFlow implementation AFAICT).

We want to keep the code simple in the present repo and don't provide specific optimizations out-of-the-box but if you want to add some of bert-as-a-service optimizations, here are the most straightforward ones:
- use `with torch.no_grad()` to avoid computing gradient during evaluation (should divide memory consumption by two)
- use fp16 code to get another factor of 2, probably the easiest is to use the amp wrapper in NVIDIA's apex library, see the details [here](https://github.com/NVIDIA/apex).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
