Compressive Transformer
# 🌟 New model addition

## Model description

<table>
    <tr><th>Title</th><td>Compressive Transformers for Long-Range Sequence Modelling (ICLR '20)</td></tr>
    <tr><th>arXiv</th><td><a href="https://arxiv.org/pdf/1911.05507.pdf">1911.05507</a></td></tr>
    <tr><th>Blog</th><td><a href="https://deepmind.com/blog/article/A_new_model_and_dataset_for_long-range_memory">A new model and dataset for long-range memory</a></td></tr>
</table>

__Compressive Transformer__ is an attentive sequence model which __compresses past memories__ for long-range sequence learning. The idea is similar to [Transformer-XL](https://arxiv.org/pdf/1901.02860.pdf). However, the memories are compressed in __Compressive Transformer__, making it leverage longer past memories compared to __Transformer-XL__.

## Open source status

- [ ] the model implementation is available
- [ ] the model weights are available
- [ ] who are the authors
Interested in model weights too but currently not available. Author does mention releasing tf code here:

https://news.ycombinator.com/item?id=22290227

Requires tf 1.15+ and deepmind/sonnet ver 1.36. Link to python script here:

https://github.com/deepmind/sonnet/blob/cd5b5fa48e15e4d020f744968f5209949ebe750f/sonnet/python/modules/nets/transformer.py#L915

Have tried running as-is but doesn't appear to have options for training on custom data as per the paper and available data sets.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
