1. Checking on other gpus

image371×1163 7.25 KB

Since I found this error on sunday, I mostly checked on the gpus I can access on sunday.
(This is the capture report I wrote for bugs issues to send which I tried on sunday, checked on gpu - 0, 1, 3, 4, 5, 6, 7)
It’s some kinda weird that sometimes it’s okay but sometimes it’s making weird behaviors.
(See gpu 0 and 6 - sometimes it’s okay but sometimes it makes nan)
And still I can find the same issues on gpu - 0, 1, 3, 4, 6
(cause I’m using gpu server with other users, it’s quite hard to check on all gpus… but I checked as much as I can)
Also, some trials go well but some trials make nan behaviors.
For more details, I upload my debug codes on google drive links with open access!
I run my codes using jupyter so u can check the logs that I printed.
https://drive.google.com/drive/folders/1eQavapn1NQ_NQkU6Kg84WX42UYy7CnSD

2. Specific updates
Actually I’m using gpu server as rent, I don’t know the owner did some special updates or not…
I know it could be hardware issues and also the software issues so I asked for gpu server owner for this issues.
Also I’m not the only one who is using gpu server, that might be the reason?? (like there are more than 2 processes are working per 1 gpu for every 8 gpus)
If you need more details, I’ll try to reproduce it or explain I can do