Hi everyone,
I have been witnessing some unstable training in a large network in a P100. I managed to nicely train the network a couple of months ago but, after an update of the NVIDIA drivers to v387.26, I’ve been finding some unstable training in PyTorch, with losses going down much slower than before and getting validation losses with large oscillations.
Around that time I also updated the PyTorch version to 0.3.0, so at the beginning I thought it was related with the version of PyTorch and some subtle API change. However, after many tests, I think I have isolated the problem to the change in the version of the NVIDIA drivers because the training (with PyTorch 0.3 and the very same code) is smooth when using a Titan X with NVIDIA drivers 367.48.
Here are the details of the system, but the results are independent of the CUDA version (it also happens in CUDA 9.0)
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:02:00.0 Off |                  Off |
| N/A   30C    P0    24W / 250W |     29MiB / 12193MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

Has anyone witnessed this kind of instability for this specific version of the drivers and Pytorch?
Thanks!