After some days, I managed to find the time to run the very same code on v0.2.0 and I don’t find these instabilities. This happens not only on the P100 but also on a Titan X, where training the same network both in v0.3.0 and v0.2.0 give completely different losses and performances (with 0.3.0 being much worse). I will try to make some tests to see if I can isolate my problem. A priori I don’t know what could be happening because my networks are pretty simple ones using conv2d, batchnorms and relus.