The library provides a version of the model for language modeling only.
Reformer
Reformer: The Efficient Transformer, Nikita Kitaev et al .
An autoregressive transformer model with lots of tricks to reduce memory footprint and compute time. Those tricks
include:
Use Axial position encoding (see below for more details). It’s a mechanism to avoid
  having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller
  matrices.
Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more
  details). It's a technique to avoid computing the full product query-key in the attention layers.
Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during
  the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them
