Create smaller number of heads in attn without pruning using shared parameters
# 🚀 Feature request

Instead of prunning heads or masking heads, create new linear layers with views of the larger linear layers that contains all heads.

## Motivation

For various models (Bert, Distilbert, etc.) I would like to be able to experiment on using separate heads without having to prune the heads. For example, training layers with separate heads. There is a way to masks heads, but the computation is still performed over all of the k, q, v tensors, and then masking is performed. 

Is there a way to create views of the heads that you would otherwise prune, so that the extra computation per unused heads is not performed. My understanding is that the attention computation is one of the more expensive operation, since it is quadratic in the seq_len. 

For example, with DistilBert, I was thinking of replacing, q_lin, v_lin, and k_lin with a shared linear layer that is a view into the original q_lin, k_line, v_lin.  

## Contribution

I am thinking of adapting this code from prune_linear_layer from modeling_utils.py. Do you think this will work? Is there a better way to do this?
``
def share_linear_layer_by_index(layer, index, dim=0):
    """ Create a new linear layer (a model parameters) with shared parameters from index of the old layer
        Return new layer with requires_grad=True.
    """
    index = index.to(layer.weight.device)
    W = layer.weight.index_select(dim, index)
    if layer.bias is not None:
        if dim == 1:
            b = layer.bias
        else:
            b = layer.bias[index]
    new_size = list(layer.weight.size())
    new_size[dim] = len(index)
    new_layer = nn.Linear(new_size[1], new_size[0], bias=layer.bias is not None).to(layer.weight.device)
    new_layer.bias.requires_grad = False
    new_layer.weight = W
    new_layer.weight.requires_grad = True
    if layer.bias is not None:
        new_layer.bias.requires_grad = False
        new_layer.bias = b
        new_layer.bias.requires_grad = True
    return new_layer

``
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
