GPT-2 XL PyTorch Quantization for use on a Cloud Server
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to run a fast (well, relatively) and interactive version of GPT-2 XL on an Ubuntu 18.04 Cloud Server using python. I have no intention of using the model for anything other than giving it a prompt and getting a generated response out of it.

I know that quantized models are usually used for mobile devices, but I want to use it on a server. Using a python script from a [huggingface tutorial](https://towardsdatascience.com/on-device-machine-learning-text-generation-on-android-6ad940c00911), I was able to convert the tensorflow version of GPT-2 small and medium over to `.tflite` files. When I tried to convert GPT-2 Large however, I ran into the same memory error as [here](https://github.com/huggingface/tflite-android-transformers/issues/4). There was an answer to a semi-related [stack overflow](https://stackoverflow.com/a/36358913) question which suggested looping through the data to be quantized, but I couldn't figure out how to apply this method to GPT-2. I suspect it might be able to be done by looping through the decoder layers and merging them afterwards.

In any case, I then moved onto the PyTorch versions of the models (Thank you so much by the way for providing these!). PyTorch recently released support for quantizing models. I've been trying to adapt the [BERT quantization tutorial](https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html#apply-the-dynamic-quantization) to GPT-2, but I keep getting the error `RuntimeError: Could not run 'aten::quantize_per_tensor' with arguments from the 'CUDATensorId' backend. 'aten::quantize_per_tensor' is only available for these backends: [CPUTensorId, VariableTensorId].`. Here's a code snippet:

```
def text_generator(
    text="",
    quiet=False,
    nsamples=1,
    unconditional=None,
    batch_size=-1,
    length=-1,
    temperature=0.7,
    top_k=40,
):
    if os.path.exists("bin/gpt2-large-pytorch_model.bin"):
        state_dict = torch.load(
            "bin/gpt2-large-pytorch_model.bin",
            map_location="cpu" if not torch.cuda.is_available() else None,
        )
    else:
        print("Please download gpt2-pytorch_model.bin and/or place in bin folder")
        sys.exit()

    if batch_size == -1:
        batch_size = 1
    assert nsamples % batch_size == 0

    seed = random.randint(0, 2147483647)
    np.random.seed(seed)
    torch.random.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    print("CUDA AVAILABILITY: {}".format(torch.cuda.is_available()))
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # Load Model
    enc = get_encoder()
    config = GPT2Config(
        vocab_size_or_config_json_file=50257,
        n_positions=1024,
        n_ctx=1024,
        n_embd=1280,
        n_layer=36,
        n_head=20,
        layer_norm_epsilon=1e-5,
        initializer_range=0.02,
    )
    model = GPT2LMHeadModel(config)
    model = load_weight(model, state_dict)
    model.share_memory()
    model.to(device)
    model.eval()
    print(model)
    quantized_model = torch.quantization.quantize_dynamic(
        model, {torch.nn.Linear}, dtype=torch.qint8
    )
    print(quantized_model)
```

This program is a slightly modified version of graykode's repo [here](https://github.com/graykode/gpt-2-Pytorch/blob/master/main.py). Is there a way for me to quantize the PyTorch version of GPT-2, or is as of now impossible?
I found that by changing `device = torch.device("cuda" if torch.cuda.is_available() else "cpu")` to `device = torch.device("cpu")` the program was able to continue, except the quantized models are larger for some reason...

|             | Old Size | New Size |
|-------------|----------|----------|
| small       | 548.1MB  | 586.7MB  |
| medium      | 1.5GB    | 1.6GB    |
| large       | 3.2GB    | 3.3GB    |
| extra large | 6.4      | 6.5      |
<|||||>In the line where I quantize the model (`quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)`), swapping out `torch.nn.Linear` for `torch.nn.Bilinear` works better, except the file size is still the same as the unquantized model. To that extent, performance is also worse than the unquantized model.

I tried swapping out `qint8` for `float16` but I just got similar results.<|||||>I'm in the same boat, here is my script:

```from __future__ import absolute_import, division, print_function

import logging
import numpy as np
import os
import random
import sys
import time
import torch

from argparse import Namespace
from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
                              TensorDataset)
from tqdm import tqdm
from transformers import (GPT2Config, GPT2Model, GPT2Tokenizer,)
from transformers import glue_compute_metrics as compute_metrics
from transformers import glue_output_modes as output_modes
from transformers import glue_processors as processors
from transformers import glue_convert_examples_to_features as convert_examples_to_features

# Setup logging
logger = logging.getLogger(__name__)
logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s -   %(message)s',
                    datefmt = '%m/%d/%Y %H:%M:%S',
                    level = logging.WARN)

logging.getLogger("transformers.modeling_utils").setLevel(
   logging.WARN)  # Reduce logging

print(torch.__version__)

"""We set the number of threads to compare the single thread performance between FP32 and INT8 performance. In the end of the tutorial, the user can set other number of threads by building PyTorch with right parallel backend."""

torch.set_num_threads(1)
print(torch.__config__.parallel_info())

configs = Namespace()

# The output directory for the fine-tuned model.
configs.output_dir = "./pytorch_models/pytorch-openai-transformer-lm/model"


# The model name or path for the pre-trained model.
configs.model_name_or_path = "pytorch_model.bin"
# The maximum length of an input sequence
configs.max_seq_length = 128

configs.task_name = "MRPC".lower()
configs.processor = processors[configs.task_name]()
configs.output_mode = output_modes[configs.task_name]
configs.label_list = configs.processor.get_labels()
configs.model_type = "bert".lower()
configs.do_lower_case = True

# Set the device, batch size, topology, and caching flags.
configs.device = "cpu"
configs.per_gpu_eval_batch_size = 8
configs.n_gpu = 0
configs.local_rank = -1
configs.overwrite_cache = False

# Set random seed for reproducibility.
def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
set_seed(42)

model = GPT2Model.from_pretrained(configs.output_dir)
model.to(configs.device)

quantized_model = torch.quantization.quantize_dynamic(
    model, dtype=torch.qint8
)

quantized_output_dir = configs.output_dir + "quantized/"
if not os.path.exists(quantized_output_dir):
    os.makedirs(quantized_output_dir)
    quantized_model.save_pretrained(quantized_output_dir)
```<|||||>Could you surround your code in triple tick marks to make your code more readable?

Microsoft has apparently open sourced a [distilled variant of GPT-2](https://github.com/microsoft/DialoGPT) designed for conversations. It's based off of Huggingface's work [here](https://github.com/huggingface/transfer-learning-conv-ai) and has the option of being trained in FP16, which sounds promising.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Were you able to get a quantized version of GPT-2?<|||||>I wasn't. Turns out there is a operation that is not supported by tensorflow yet. I don't remember what because it was a time ago.  Gave up on the project. Sorry if this isn't very useful. Just updating.<|||||>I unfortunately wasn't able to create/find a quantized model either. I just ended up using the full XL Model instead.<|||||>I managed to quantize Pytorch GPT-2 XL to int8 with `quantize_torch_model` method from [this script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/quantize_helper.py). As easy as:
```model = QuantizeHelper.quantize_torch_model(model)```

After that I observed 4x speedup on CPU (and changes in predicted scores).

You might also want converting it to torchscript with
```inference = torch.jit.trace(model, input_ids)```

You can find the complete usage example [here](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/benchmark.py). <|||||>Thank you for updating. I perused the code and I saw a condition which requires CPU for INT8 and GPU for FP16. Is the INT8 model runnable on GPU? <|||||>@omeysalvi I didn't test in8 version on GPU, but these benchmarks even avoid this combination, so I guess it is not something very promising to run int8 on GPU.<|||||>@omeysalvi just tried int8 on GPU — doesn't work. Error:
`RuntimeError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDATensorId' backend. 'quantized::linear_dynamic' is only available for these backends: [CPUTensorId].`

I guess fp16 is generally recommended for optimizing GPT2 on GPU.<|||||>Hi @klimentij. Thanks for the method.
I have tried it with the following code, and it works very well
```
model = model_class.from_pretrained(model_name_or_path)
model = QuantizeHelper.quantize_torch_model(model)
model.to(device)
```
but if I try to save the quantized model and reload it by
```
model.save_pretrained(quantized_model_path)
model = model_class.from_pretrain(quantized_model_path)
model.to(device)
```
the saved qunatized model size is about half of the initial model as it only quantized Conv1D/Linear layer.
But the quantized model generated very strange results which has none sense...
Do you know any possible reason?
<|||||>@carter54 for my purposes int8 generation quality was not acceptable (but not nonsense, more like from GPT2-small), so I didn't even try to save it. If you're okay with generation quality after quantization, I'd try saving it using other means (e.g. torch native saving or even pickling), avoiding `save_pretrained`.<|||||>@klimentij Thanks mate, I will have a try.<|||||>@carter54  I also run into the same problem, the model is doing well before saved and loaded with `save_pretrained`. Inference using saved and loaded quantized model gives 50% less F1 score. Have you tried what @klimentij suggested? Mind to share the result? Thanks in advance.<|||||>I had issues with klimentij's suggestion but I solved it by extracting the `conv1d_to_linear` functions. I had to load a previous model into a pretrained version of GPT2 so ignore that part if you don't have to do it.
```python
def _conv1d_to_linear(module):
    in_size, out_size = module.weight.shape
    linear = torch.nn.Linear(in_size, out_size)
    linear.weight.data = module.weight.data.T.contiguous()
    linear.bias.data = module.bias.data
    return linear


def conv1d_to_linear(model):
    """in-place
    This is for Dynamic Quantization, as Conv1D is not recognized by PyTorch, convert it to nn.Linear
    """
    for name in list(model._modules):
        module = model._modules[name]
        if isinstance(module, Conv1D):
            linear = _conv1d_to_linear(module)
            model._modules[name] = linear
        else:
            conv1d_to_linear(module)


tokenizer = GPT2TokenizerFast.from_pretrained("gpt2-xl")
text = "Test Text."
tokens = tokenizer(text, return_tensors="pt")["input_ids"]
model = torch.load("../model.pt")
model.resize_token_embeddings(len(tokenizer))
model.eval()
pretrained_model = GPT2LMHeadModel.from_pretrained("gpt2-xl", torchscript=True)
pretrained_model.resize_token_embeddings(len(tokenizer))
pretrained_model.load_state_dict(model.state_dict())
pretrained_model.eval()
conv1d_to_linear(pretrained_model)
quantized_model = torch.quantization.quantize_dynamic(
    pretrained_model, {torch.nn.Linear}, dtype=torch.qint8
)
traced_model = torch.jit.trace(quantized_model, tokens)
torch.jit.save(traced_model, "quantized_traced_model.pt")


def print_size_of_model(model):
    torch.save(model.state_dict(), "temp.p")
    print("Size (MB):", os.path.getsize("temp.p") / 1e6)
    os.remove("temp.p")


print_size_of_model(pretrained_model)
print_size_of_model(quantized_model)
```<|||||>> I managed to quantize Pytorch GPT-2 XL to int8 with `quantize_torch_model` method from [this script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/quantize_helper.py). As easy as:
> `model = QuantizeHelper.quantize_torch_model(model)`
> 
> After that I observed 4x speedup on CPU (and changes in predicted scores).
> 
> You might also want converting it to torchscript with
> `inference = torch.jit.trace(model, input_ids)`
> 
> You can find the complete usage example [here](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/benchmark.py).

Hi @klimentij,

I am able to use the QuantizeHelper class to convert Conv1D to Linear layer (in `DistilGPT2`) and thereby using it to quantize.

The problem I face now is, them the last `lm.head` layer is being quantized it converts
`lm_head.weight`

to

`lm_head.scale
lm_head.zero_point
lm_head._packed_params.weight
lm_head._packed_params.bias`

Now, the quantized params in the layer `lm_head._packed_params.bias` is just `None`.

What shall be done in this case?<|||||>following<|||||>> @carter54 I also run into the same problem, the model is doing well before saved and loaded with `save_pretrained`. Inference using saved and loaded quantized model gives 50% less F1 score. Have you tried what @klimentij suggested? Mind to share the result? Thanks in advance.

I also tried to quantize the GPT2 model but the generated text is really not good. I did some research and find the quantization aware training could be a solution, but it requires implementing the GPT2 model from scratch.