Do you think that error is the source of the illegal memory access errors in AMP training too? Why would that one only happen when the number of discriminators is set to the (default) of 2? I’m happy to use 1, 3, or 4 discriminators, as long as it can train without getting those gradient overflow, NaN, memory access errors.
I installed the latest version using !pip install torch===1.5.0 torchvision===0.6.0 -f https://download.pytorch.org/whl/torch_stable.html.
But now I get an error trying to install apex:
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-4rw655gw/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-4rw655gw/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-g9ds8p5h/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
And when I try to run the training script, I get a new error too:
Traceback (most recent call last):
  File "train.py", line 17, in <module>
    opt = TrainOptions().parse()
  File "/content/pix2pixHD/options/base_options.py", line 80, in parse
    torch.cuda.set_device(self.opt.gpu_ids[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 245, in set_device
    torch._C._cuda_setDevice(device)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 149, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 63, in _check_driver
    of the CUDA driver.""".format(str(torch._C._cuda_getDriverVersion())))
AssertionError: 
The NVIDIA driver on your system is too old (found version 10010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'train.py', '--local_rank=0', '--fp16', '--resize_or_crop', 'scale_width_and_crop', '--name', 'label2city_512p', '--label_nc', '0', '--no_instance']' returned non-zero exit status 1.

Here’s a colab: https://colab.research.google.com/drive/1ed3lxHDORKkubREnIbnRiWqbghDF5Kju?authuser=5#scrollTo=vzISEGB8Da_H 2