capability of equipping the linear self-attention with relative position encoding. As a result, the enhanced
transformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We
release the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing
experiment for English benchmark will soon be updated.
Tips:
RoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown
  improved performance on classification tasks with long texts.
This model was contributed by junnyu. The original code can be found here.
RoFormerConfig
[[autodoc]] RoFormerConfig
RoFormerTokenizer
[[autodoc]] RoFormerTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
