Memory leakage with bert-large-uncased-whole-word-masking-finetuned-squad
# 🐛 Bug

## Information
I am using 'bert-large-uncased-whole-word-masking-finetuned-squad'  and observed an memory leak  during the inference. Below the code snippet to reproduce it

tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
model = AutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")

device = 'cuda'
question = 'what is your question'
answer = 'This is the  answer'

 def tokenize_qa(tokenizer,question, answer_context):
    input_ids =  tokenizer.encode(question, answer_context, max_length=512)
    tokens =  tokenizer.convert_ids_to_tokens(input_ids)
    sep_index = input_ids.index( tokenizer.sep_token_id)
    num_seg_a = sep_index + 1
    num_seg_b = len(input_ids) - num_seg_a
    segment_ids = [0] * num_seg_a + [1] * num_seg_b
    assert len(segment_ids) == len(input_ids)
    return input_ids, segment_ids, tokens

###################### Inference Code  ##########################
input_ids, segment_ids, tokens= tokenize_qa(tokenizer,question,answer_context)
start_scores, end_scores = model(torch.tensor([input_ids], device= device),token_type_ids=torch.tensor([segment_ids], device= device))

When i run the inference code multiple times it accumulates large  gpu memory and finally runs OOM. Please check let me know if anything wrong here.

Thank you.

Language I am using the model : English,
     
- `transformers` version: '2.11.0'
- Platform: Ubuntu 18.04
- Python version:3.6.9
- PyTorch version (GPU): 1.4.0

Hi, @ashispapu, you might wanna wrap the inference code with
`with torch.no_grad():`<|||||>Thanks for the response. Should not it be grad disabled by default like other models  in transformer during inference.<|||||>I don't think so, how would the model know that you are doing training or inference. You can use `pipeline` for question answering, that takes care of such things 