 chunked T5 - small (cT5-small)  Github: https://github.com/mtreviso/chunked-t5  A T5 model that uses a new loss where a special end-of-chunk token `</c>` is appended after sentinel tokens.  The decoder has to predict the full input with masked tokens followed by `</c>`.  This allows a much faster auto-regressive generation since the decoder can predict multiple tokens in parallel.  For example, for the input `the quick brown fox jumps over the lazy dog`: ``` encoder: the <extra_id_0> fox jumps <extra_id_1> the lazy dog  T5 decoder : <extra_id_0> quick brown <extra_id_1> over <extra_id_2> cT5 decoder: <extra_id_0> quick brown </c> <extra_id_1> over </c> <extra_id_2> ```  The generation may look like this for T5 and cT5: ``` T5: <extra_id_0> T5: <extra_id_0> quick T5: <extra_id_0> quick brown T5: <extra_id_0> quick brown <extra_id_1> T5: <extra_id_0> quick brown <extra_id_1> over T5: <extra_id_0> quick brown <extra_id_1> over <extra_id_2> T5: <extra_id_0> quick brown <extra_id_1> over <extra_id_2> </s>  cT5: <extra_id_0> <pad> <extra_id_1> <pad> <extra_id_2> </s> cT5: <extra_id_0> quick <pad> <extra_id_1> over <pad> <extra_id_2> </s> cT5: <extra_id_0> quick brown <pad> <extra_id_1> over </c> <extra_id_2> </s> cT5: <extra_id_0> quick brown </c> <extra_id_1> over </c> <extra_id_2> </s> ```  In the original T5, the decoder is called \\(n_s + 1 + \sum_i |s_i|\\) times autoregressively,  where \\(n_s\\) is the number of sentinel tokens and \\(s_1,...,s_{n_s}\\) are the predicted chunks.  In contrast, cT5's decoder is called just \\(max_i |s_i| + 1\\) times.  The generation stops when all sentences were fully translated to complete chunks, i.e., until all `</c>` tokens were generated.  Alternatively, you can also set `max_chunk_size` to manually force the model to stop after generating a chunk with `max_chunk_size` tokens. The overhead of calling the decoder with a longer input is less pronounced since this computation can be parallelized in GPUs/TPUs.  