By moving all variable tensors and models into gpus[0](not gpu 0), the first model replica will be on the right device you selected(gpus=[xx, xx, xx]) even though the broadcast function will still ignore it.