(1) The GPUs are all nvidia P100, with driver 390.46.
(2) Problem only occurs at the initialization part, model = model.to_cuda()。The time taken for each training and evaluating iteration is normal。
(3) It seems that the latest version of pytorch (v0.4.1) requires cuda 9.0 or 9.2. I’m not sure if we can successfully compile it with CUDA8. Maybe we can try it.
By the way, if you have experience in building a pytorch wheel containing all dependencies, just like the official whl, could you please share it to us? Or tell us where to find the solutions. It will be very helpful to us.