Autoregressive differentiable decoding? (no teacher forcing nor self-reconstruction)
# 🚀 Feature request

Hi, 

Is there any way to perform autoregressive differentiable decoding? 
As far as I know, for **encoder-decoder models** (e.g., T5, BART), we have the following:
- **.forward()** performs decoding using **teacher forcing**, or in the case of BART, shifts the _input_ids_ to the right if no _decoder_input_ids_ are given in order to do self-reconstruction. In that case the decoding is differentiable but not autoregressive as it uses teacher forcing (or some kind of supervision). 
- **.generate()** enables decoding using **greedy search**, **beam search**, or **top-k sampling**. In that case, the decoding is autoregressive but not differentiable. 

I would like to **_decode differentiably_** by using **plain autoregression**: at each decoding step t, we feed to the decoder the token generated at previous step t-1. By _differentiably_, I mean that we don't take the argmax() to select a single generated token. Rather, we use embeddings weighted by the logits, to get a pseudo-token that we can feed to the decoder at the next step. 
Such a technique of using weighted averages of embeddings as pseudo-tokens has been used in recent research:
https://arxiv.org/abs/1905.05621

We can implement this manually using a for loop over decoding steps calling .forward() at each step, but it is quite slow. 

## Motivation

The motivation behind this feature request is to be able to generate sequences differentiably without supervision, and fast, which can be very useful for several research purposes. 


As far as I know, in generation phase, the autoregressive decoding procedures are all implemented in each searching method such as greedy search, meaning that the output embeddings will certainly be sampled by `argmax` or other sampling methods. Besides, the autoregressive decodings are all running in loop statement currently.

I think that implementing a method that solely performs autoregression and makes the output differentiable seems great and feasible, but if the bottleneck in your code is loop statement, this way may have no significant help in performance.<|||||>cc @patrickvonplaten <|||||>Yeah currently gradient backprop is not really supported in `transformers` sadly. Think it would require some major changes to implemented this. Feel free to give it a stab! Would also be very interested in knowing how feasible this is in PyTorch!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.