fast gpt2 inference
I have a fine-tuned ```GPT2LMHeadModel``` (gpt2-medium) which I am using to run inference on large data (>60M sequences) offline. At each iteration, my input is a batch of 30 variable-length sequences which gets padded according to the max length of the batch. My current speed is around 8 secs/iter and input sequences have around 150 tokens.

I am using pytorch's ```BucketIterator``` to group sequences of similar length and minimize padding. I am using fp-16. These are increasing the throughput. The GPU I am using is Tesla V100.

Can you please suggest what all other optimizations that I can do to increase the speed?
Have given some thought to using TorchScript but since my input sequence length changes each time, the easier tracing approach won't work.

ONNX also faced the same problem.<|||||>Hi rajarsheem,
Can you please share your code that doing the batch inference with variable-length sequences?

Thanks<|||||>Is gpt2 model traceable? <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
