If you are loading the data onto the CPU (as would be the usual work flow), the number of workers should not change the usage of the GPU memory.
Could you post your Dataset and how you are using the DataLoader such that your device is running out of memory?