Are you pushing the data to the GPU inside the Dataset (in the __init__ or __getitem__)?
If so, increasing the number of workers would also increase the GPU memory usage, since each worker would push the data to the device.
If that’s not the case, could you post an executable code snippet so that we could reproduce this issue, as the device memory shouldn’t increase while loading data using the CPU.