Hello, I’m currently running an NMT experiment using the finetune.py from examples/seq2seq. With some research, I found the idea of leveraging pre-trained models instead of training from scratch. My model aims to translate pt_BR to es_ES, so my choice was to take advantage of https://huggingface.co/Helsinki-NLP/opus-mt-pt-ca 6 which seemed from very proximate domains. I’m using the opus dataset pt_BR and es with 55M sentence pairs (with a quick qualitative analysis I believe this data has low/medium quality but is a very considerable amount). After doing the first run and, as this requires a huge amount of processing and costs $, some questions came to my mind, you can check my experiment on Weights and Biases: https://wandb.ai/jpmc/data/runs/3hahd13a?workspace=user-jpmc 4 .

I’m having a huge inconsistency on the GPU Usage, as you can see from the link, I have 8 Tesla K80, and tried some batch_sizes that crashed earlier, while this one (32) crashed after 41h of training. Then, when lowered to 16, it crashed even earlier (check the system information in the other run from this experiment).
One epoch is estimated to take 60hours, is this correct? I have no experience in this problem with transformers.
The val_BLEU seems to drop but not the val_loss, is it possible that the bleu value is not being logged correctly? What could cause this?
This is my first time using the library, any advice on models and hyper-parameters I could tune? Should I train from scratch?