N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models
while being much more memory-efficient and much faster on long sequences.
This model was contributed by patrickvonplaten. The Authors' code can be
found here.
Note:
Reformer does not work with torch.nn.DataParallel due to a bug in PyTorch, see issue #36035
Axial Positional Encodings
Axial Positional Encodings were first implemented in Google's trax library
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embedings vector of size \(d\) being the config.hidden_size for
every position \(i, \ldots, n_s\), with \(n_s\) being config.max_embedding_size. This means that having
a sequence length of \(n_s = 2^{19} \approx 0.5M\) and a config.hidden_size of \(d = 2^{10} \approx 1000\)
