There seem to be multiple issues based on your description:

if you are seeing a memory increase by running the script, this points to the common issue of storing tensors, which are attached to the computation graph, and thus PyTorch won’t be able to free it. Often this is done by e.g. appending the output or loss tensors to a list without detaching.
if you are seeing the same error message right after restarting the machine, your OS might use GPU memory and thus not the complete GPU is available to PyTorch (if you have a monitor plugged into the GPU, it’ll use memory for the display output)
if you are seeing a clean GPU at the beginning and see an increased memory usage with a DataLoader and multiple workers only, please post a code snippet to reproduce this issue.