Option 2 would be the standard approach, i.e. the loss would be calculated on the default device.
This also means that your default device will accumulate more memory then the other GPUs due to the target, loss etc. being stored there.
Option 1 is described in @Thomas_Wolf’s blog post 20.
Your approach sounds valid to me. Batching dicts should also work without a problem.