pytorch camembert masked-lm arxiv:2004.05150 transformers fill-mask
.gitattributes README.md config.json pytorch_model.bin sentencepiece.bpe.model special_tokens_map.json tokenizer_config.json
This model is an "extension" of [CamemBERT-base model] (https://camembert-model.fr/) to support up to 4096 tokens.
The architecture was previously extended using [AllenAI's code](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) for the procedure descriped in the [Longformer paper](https://arxiv.org/abs/2004.05150). [Their code](https://github.com/allenai/longformer) is Apache-2.0 licensed.
Longformer is an "extension" of the RoBERTa model and CamemBERT use the same architecture as RoBERTa (they don't have same tokenizers though and CamemBERT use Whole Word Masking). 
Hence, this "long CamemBERT" can be pretained or directly fine-tuned using AllenAI's code without substantial modifications after being extended. 

I created [a notebook] (https://colab.research.google.com/drive/1B7Gj32yqM2NETdnOlqt997RArYJwV24D#scrollTo=5VKUZnBb4vvD) in order to see the model in action with a HuggingFace pipeline (fillmask).
The model can be pretrained using FP16, or using TPUs with TensorFlow or PyTorch-XLA.