I’m training a SSD model, one runing on a V100s, the other running on 1080ti. And I have set up a target mAP e.g., to achieve at 20% and keep a log of how many epochs it took to arrive at that target mAP. I also keep all the settings the same(e.g, batch size, RNG seeds) for V100s and 1080ti. But it seems that for my V100s, it’s quite steady to take 100 epochs to hit the target mAP. For my 1080ti, some times it takes more than 100 epochs, sometimes it does take 100 epochs to hit the target mAP.
So, I just want to analyze what are the possible reasons for this and what are some methods to verify the reasonings.
Here is what I think:
V100s and 1080ti are completely different series Nvidia GPU cards, they have different CUDA computability, one is 7+, the other is 6+. The architectures are also different. So there might be some algorigthms implemented diffferntly for them. The minute difference(I guess for the same CUDA op, the output shouldn’t be too different if given same inputs.) in all the ops in the model might eventually add up to make a difference?
Methods to verify this?
I think I could use the forward/backward hooks to record the intermediate results including parameters, layer outputs, and grads wrt inputs and weights. And try to figure out when the divergence happens. But the dumped file is quite large and layer results are not in the same magitude, I easily get lost. Any good practice for this?
If it’s not for the architecture reason, what are some other possible reasons? And any methods to verify? Thanks.