decoding is slow for auto-regressive decoder because the tokens are generated one at a time. I can think of only two things to improve speed for generation.

put the model in fp16 (not sure if GPT-2 works with fp6, haven’t tried myself).
See if you can use onnx to speed up inference. Maybe this 63 will help.