--source_lang en --target_lang ro \
--fp16 --sharded_ddp simple
Notes:
This feature requires distributed training (so multiple GPUs).
It is not implemented for TPUs.
It works with --fp16 too, to make things even faster.
One of the main benefits of enabling --sharded_ddp simple is that it uses a lot less GPU memory, so you should be
  able to use significantly larger batch sizes using the same hardware (e.g. 3x and even bigger) which should lead to
  significantly shorter training time.
To use the second version of Sharded data-parallelism, add --sharded_ddp zero_dp_2 or --sharded_ddp zero_dp_3 to the command line arguments, and make sure you have added the distributed launcher -m torch.distributed.launch --nproc_per_node=NUMBER_OF_GPUS_YOU_HAVE if you haven't been using it already.
For example here is how you could use it for run_translation.py with 2 GPUs:
