Hi,
May it be, you are doing some image transforms on gpu? As transforms are usually made by dataloader if you moving data to gpu somethere in your dataset code, increasing the number of workers will increase the number of dataloader replicas and so allocated gpu memory will be increased as well.