I am trying to parallelize an existing model (Transformer) on muliple GPUs. I have a NVIDIA DGX-1 with 8 Volta.
These are the issues/questions I ran into:

not clear which value should I set for model.to(device). currently trying with simply .to(‘cuda’)
the model is using a lambda LR scheduler but with GPUs > 1 it exits with an execution error “UnboundLocalError: local variable ‘values’ referenced before assignment”
how can I effectlvely check tensor boundaries limit? I am getting the impression that moving a model from 1 to multiple GPUs requires a profound refactoring of the code. It is extremely difficult to understand where indexs can go out of bounds.

Thanks