@ptrblck  also see an issue with the sparseAdam optimizers (I have a sparse embedding in my model) with apex, is there a way to solve that ?
optimizers = [torch.optim.SparseAdam(params=[list(model.parameters())[0]], lr=lr),
                              torch.optim.Adam(params=list(model.parameters())[1:], lr=lr, amsgrad=True) ]


with amp.scale_loss(loss, optimizers) as scaled_loss:
                    scaled_loss.backward()

The error :

scaled_loss.backward()

File “C:\Users\S\Anaconda3\envs\pytorch\lib\contextlib.py”, line 119, in exit
next(self.gen)
File “C:\Users\S\Anaconda3\envs\pytorch\lib\site-packages\apex\amp\handle.py”, line 123, in scale_loss
optimizer._post_amp_backward(loss_scaler)
File “C:\Users\S\Anaconda3\envs\pytorch\lib\site-packages\apex\amp_process_optimizer.py”, line 241, in post_backward_no_master_weights
post_backward_models_are_masters(scaler, params, stashed_grads)
File “C:\Users\S\Anaconda3\envs\pytorch\lib\site-packages\apex\amp_process_optimizer.py”, line 120, in post_backward_models_are_masters
scale_override=grads_have_scale/out_scale)
File “C:\Users\S\Anaconda3\envs\pytorch\lib\site-packages\apex\amp\scaler.py”, line 117, in unscale
1./scale)
File “C:\Users\S\Anaconda3\envs\pytorch\lib\site-packages\apex\multi_tensor_apply\multi_tensor_apply.py”, line 30, in call
*args)
RuntimeError: sparse tensors do not have is_contiguous (is_contiguous at …\aten\src\ATen\SparseTensorImpl.cpp:55)
(no backtrace available)