Here is some more info. Our person in IT did some more testing (he is great!).

just to be sure, I went on AWS and ran it on one of the new GPU systems with Volta GPUs. I used the deep-learning AMI with pytorch already setup and with CUDA 9.0, Python 3.6.6, nvidia driver
396.37 and pytorch version 0.4.0 and 0.4.1 - both failed with even much worse issues than on our system (the last run was:  cuda (9079, 10921) and that was the best result!!) It was also quite slow with the GPU version. At this point I am willing to rule out hardware setup or build issues on our system and think that either it is some kind of pytorch problem, or there is some potential issue with the code. I admit that I have not had a chance to look at the code yet, but will try to get to it over the next few days. The AMI that I used was:
https://aws.amazon.com/marketplace/pp/B077GCH38C

This seems to point to a general problem now not just on our systems.