Faster mask computation 
# 🚀 Feature request

Currently, the masking is done using the full prediction matrix which is both memory and computation inefficient. One example is in [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_albert.py#L675). I think with Pytorch indexing, first and foremost, we don't need to do full mask matrix construction. Second, it can be much faster. Fairseq does a similar thing in [here](https://github.com/pytorch/fairseq/blob/cce6dcb1cca85955a82879ea5064fe8202e8f412/fairseq/models/roberta/model.py#L217)

## Motivation

If input is [n,m], currently the code creates a clone of [n,m] mask where non-masked inputs are -100. Then it does a full output projection: if input in [n, m] and hidden representation is [n, m, h], the final output will be a huge [n, m, v] if v is the vocabulary size. Instead we can think of mask indices of size k<< n* m, and thus the we can extract a [k, h] submatrix from [n, m, h], then can have a smaller output result [k, v].

## Your contribution
This a sample code (I extracted from a bigger code of mine; sorry if some variables are not defined in the snippet):
```py
mask_prob = 0.15
mask = torch.empty(input_ids.size()).uniform_(0, 1) < mask_prob
mask[pads] = False  # We should not mask pads.
masked_ids = input_ids[mask]
replacements = masked_ids.clone()
for i in range(len(replacements)):
    r = random.random()
    if r < 0.8:
        replacements[i] = mask_id
    elif r < 0.9:
        # Replace with another random word.
        random_index = random.randint(0, vocab_size - 1)
        replacements[i] = random_index
    else:
        # keep the word
        pass
input_ids[mask] = replacements

text_hidden, text_cls_head = albert_model(texts, attention_mask=pads)
masked_hidden_state = text_hidden[mask]
output_predictions = F.log_softmax(albertMLMHead(masked_hidden_state), dim=1)
```

Hi @rasoolims ,

That sounds reasonable what you are saying! I wanted to take a look into masking optimization in a couple of weeks. If you feel like it, you could also open a PR and we take a look together :-) <|||||>Feel free to open a PR for this :-) Closing for now