I tested the original code with the simple label2city demo using apex and discovered a few things:

It runs for a few epochs but quickly has the same error. It prints “Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale…” around 10 times per epoch, until the 7th epoch, where it gives me the same CUDA error: “an illegal memory access was encountered”
It happens after 3 epochs if I set --resize_or_crop scale_width_and_crop. It scales the width by default, so I don’t know why cropping the training images would cause the error to happen more quickly.
No difference if I add --label_nc 0 --no_instance which tells it to just map images in train_A to train_B, like I do in my code.
If I combine A-to-B method with cropping using --resize_or_crop scale_width_and_crop --label_nc 0 --no_instance, I get a different error:

Traceback (most recent call last):
  File "train.py", line 85, in <module>
    with amp.scale_loss(loss_G, optimizer_G) as scaled_loss: scaled_loss.backward()                
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR (findAlgorithms at /pytorch/aten/src/ATen/native/cudnn/Conv.cpp:551)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7f8016ad4536 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xf38e75 (0x7f8017e68e75 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xf2c07a (0x7f8017e5c07a in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xf2cd91 (0x7f8017e5cd91 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xf30dcb (0x7f8017e60dcb in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: at::native::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0xb2 (0x7f8017e61322 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xf97e40 (0x7f8017ec7e40 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xfdc6d8 (0x7f8017f0c6d8 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #8: at::native::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x4fa (0x7f8017e629ba in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #9: <unknown function> + 0xf9816b (0x7f8017ec816b in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0xfdc734 (0x7f8017f0c734 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)
frame #11: <unknown function> + 0x2c809b6 (0x7f80514629b6 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x2cd0444 (0x7f80514b2444 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #13: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x378 (0x7f805107a918 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2d89c05 (0x7f805156bc05 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #15: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x16f3 (0x7f8051568f03 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x3d2 (0x7f8051569ce2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #17: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f8051562359 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f805dca1378 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #19: <unknown function> + 0xbd6df (0x7f80686476df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #20: <unknown function> + 0x76db (0x7f80697296db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #21: clone + 0x3f (0x7f8069a6288f in /lib/x86_64-linux-gnu/libc.so.6)


That error can be resolved by changing the number of discriminators from the default of 2, adding the option --num_D 1 or --num_D 3 or --num_D 4. No idea why this is, I was just testing how the number of discriminators affect output quality in non-apex training.

So it happens even on the simple demo with the default settings. But my training images make it happen immediately, maybe because their demo only has 8 training samples while I have a few thousand. But even with only 8, it happens in a few seconds, after 3-10 epochs.