jplu/tf-xlm-roberta-large showing random performance
# 🐛 jplu/tf-xlm-roberta-large showing random performance on tpu model training

## i have observed several times that jplu/tf-xlm-roberta-large showing random performance on several experiments,for example if i train a model and get validation accuracy 61% then if i run the model again (without changing anythin or doing 0% change in code) then second i get over 96% accuracy and sometimes i observed that validation accuracy stay same for all epoch and again this same code will get me totally different train statistics next time,why is this happening?

Model I am using jplu/tf-xlm-roberta-large

Language I am using the model on (English, Chinese ,turkish,spanish etc):

The problem arises when using:
jplu/tf-xlm-roberta-large for finetuning 

The tasks I am working on is:
kaggle ongoing competition : jigsaw multilingual toxic comment classification where train set contails all english comments and 0/1(toxic/non toxic) label and validation+test data contains non English comments

## To reproduce

Steps to reproduce the behavior: 

1. just simply fork and run this kernel : https://www.kaggle.com/mobassir/understanding-cross-lingual-models and you will never get same result and probably will get bit more worst result



## Expected behavior

same model training several times should always produce close result if not exact same, the randomness is high,,sometimes model with 91% validation accuracy can get us 61% validation accuracy on next run and sometimes validation accuracy of that same model get frozen and seems like the model is not learning anything (i repeat the model name : it's tf xlm roberta large)

## Environment info
kaggle kernels/notebooks using tpu
     


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>same here, mine is torch version, though.
@mobassir94 Did you fix it?<|||||>@HenryPaik1 no
i also faced this issue in both pytorch xla and in tf tpu