CPU RAM out of memory when detach from GPU
## ❓ Questions & Help

I am using the following code to get embedding layer from BERT:
```
class BertEmbedding():
    def __init__(self, load_model=None, load_config=None, model='bert-base-uncased', max_len=512, batch_size=6):
        self.pre_trained_model = model
        self.max_len = max_len
        self.batch_size = batch_size
        self.model = BertModel.from_pretrained(self.pre_trained_model)
        self.tokenizer = BertTokenizer.from_pretrained(self.pre_trained_model)
        #self.optimizer = AdamW(params = self.model.parameters(), lr=1e-5)
    
    def create_ids(self, sentences):
        logging.getLogger("transformers.tokenization_utils").setLevel(logging.ERROR) #Disable tokenizer logs, it's really annoy
        input_ids = []
        for sen in tqdm_notebook(sentences, desc="Create Ids"):
            tmp = self.tokenizer.encode(sen)
            input_ids.append(tmp)
        input_ids = pad_sequences(input_ids, 
                                  maxlen=self.max_len, 
                                  dtype='int64', 
                                  truncating='post', 
                                  padding='post')
        return input_ids
    
    def generate(self, inputs):
        test_ids = self.create_ids(inputs)
        test_dataloader = DataLoader(torch.tensor(test_ids), batch_size=self.batch_size)
        
        embedding = []
        self.model.to(device)
        self.model.eval()
        for input_ids in tqdm_notebook(test_dataloader, desc="Generating"):
            with torch.no_grad():
                last_state = self.model(input_ids.to(device))[0]
            last_state = last_state.detach().cpu().numpy()
            embedding.extend(last_state)
        return embedding

bert_embedding = BertEmbedding(batch_size=100)

embedding = bert_embedding.generate(train.sentence.values)
```

The problem is when it generate embedding layer from model (train on GPU and detach to CPU), RAM is increasing significantly (1GB --> 30GB for a list of 25,000 arrays (512,768)). While I checked with `sys.getsizeof(embedding)` = `224208` and size of `bert_embedding` is `56` only.

If I delete both `embedding` and `bert_embedding`, RAM ~ 20GB. I think that the model is still existing. How can I optimize this for CPU RAM?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
