yeah, I first implement it use dataparallel ,but my category is more than 20000。 when pytorch gather all output to gpu0  ，GPU 0 have more graphics ram consume than others ，it limits the batch_size value，my dataset nearly 400gb large， it a very long time to wait