Hi Soohwan!



 Soohwan_Kim:

After the first iteration, the loss becomes NaN.
Another strange thing, however, is that loss is calculated normally in CPU mode.
    inputs = torch.FloatTensor(batch_size, seq_length, input_size).to(device)
...
    outputs, _, _ = transformer(inputs, input_lengths, targets)



If I am reading your code correctly, inputs (the tensor you pass into your
transformer) is uninitialized.  This could definitely cause the problems you
see.



Result in CUDA mode

tensor(10.6171, device='cuda:0', grad_fn=<MeanBackward0>)
tensor(nan, device='cuda:0', grad_fn=<MeanBackward0>)
...


Result in CPU mode

tensor(12.8601, grad_fn=<MeanBackward0>)
tensor(12.5969, grad_fn=<MeanBackward0>)
...



It is true that your cuda result becomes nan after only one optimizer.step()
iteration.  But even before your first iteration, your cuda and gpu results differ.
This could certainly be due to your uninitialized inputs.  (It could also be due
to differing random initialization of the weights in transformer, depending on
how it is constructed.)  But in any event, you should start by tracking down
this difference.  You should be able to get the results (including, among other
things, the loss) returned by your cuda computation to agree (up to some
reasonable round-off error) with those of your gpu computation, at least before
round-off error has begun to accumulate over the course of a number of
iterations.
Best.
K. Frank