Ok, I dove a bit more in to this, and found the issue. In this particular instance I am using a model library where the author unfortunately had added the batch collate function as a method to the model class. Because of this the model self is also transported to the DataLoader workers.
When the training scripts exits the DataLoader worker processes are killed too, without properly dealing with the Tensors that are copied to the worker.
I refactored all the batch collating code out of the model class and thus made it independent of the self of the model instance. This resolved the issue, exiting the script is very smooth now!
So, if anyone gets this error, it could be because your are copying the model over to the DataLoader workers without knowing it.