Reproducing squad score with TFXLMRoberta?
# ❓ Questions & Help

Hello all, 

first of all, thanks for the library that is very helpful. There have been several discussions regarding XLM-Roberta and questions answering (#3732  #3694 ). On my side, I added a TFXLMRobertaForQuestionAnswering but never reproduced a decent squad score (I was always below 50% f1). The base LM I was using was xlm-roberta-base converted to tf, or the community ones (jplu), I tried with type_vocab_size=1 or type_vocab_size=2 (in order to use segment_ids as for Bert, I did it by overwriting the create_token_type_ids_from_sequences in the tokenizer). This did not really change anything. I am using a AdamWeightDecay as for Bert finetuning and I start to believe that there is no problem with my code since I saw yesterday the PR #3812 which is basically the same code as mine. For info, I am not only training on squad, I am training on squad + mlqa. Using the exact same approach, I got very good score with Bert multilingual. Since I am a bit stuck I was wondering if someone here (from huggingface or not) managed to properly train XLMRoberta with tensorflow and get good squad results. If so, it would be super helpful for me if you can share the parameters that was used (learning rate, number of epochs, etc). 

** 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
