@ptrblck, while I’m trying 1st option using apex, I got following error.
Traceback (most recent call last):
  File "main_apex.py", line 269, in <module>
    main(args)
  File "main_apex.py", line 219, in main
    args.clip_max_norm)
  File "/home/detr/engine_apex.py", line 37, in train_one_epoch
    loss_dict = criterion(outputs, targets)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/detr/models/detr.py", line 225, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/detr/models/matcher.py", line 71, in forward
    cost_bbox = torch.cdist(out_bbox, tgt_bbox, p=1)
  File "/usr/local/lib/python3.7/dist-packages/torch/functional.py", line 732, in cdist
    return _VF.cdist(x1, x2, p, None)
RuntimeError: "cdist_cuda" not implemented for 'Half'

It seems torch.cdist function is not working with half precision.
Is there any workaround?
Can this issue be resolved in nightly version?