ZeroDivisionError: float division by zero after some epochs while training using run_mmimdb.py
I am trying to train image+test model using the run_mmiddb.py(https://github.com/huggingface/transformers/tree/master/examples/research_projects/mm-imdb). I have two classes in this task.
Initially i ran it for 1 epoch with below input parameters and it went well::
--model_name_or_path bert-base-cased \
--max_seq_length 512 \
--stride_len 112 \
--num_image_embeds 3 \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 16 \
--gradient_accumulation_steps 20 \
--patience 5 \
--fp16 \

and from log i have "Num examples = 5703" while training and in evaluation "Num examples = 1176".

Now, when i am running it for 20 epochs and reduced gradient_accumulation_steps to 6. It gave error as
"ZeroDivisionError: float division by zero"

maybe after 13 epochs i guess(I lost log some of it).
Also the loss from startig onwards is "nan". I dont know how. I see the loss fucntion as 
"criterion = nn.BCEWithLogitsLoss(pos_weight=label_weights)"
Even if loss is "nan" from starting, I see 'macro_f1' going up from 55 to 58 and again back to 55 before this above mentioned final error.
Any suggestion/solution is appreciated. Thanks
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.